[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:08 dist-test-slave-grzf krb5kdc[15741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:08 dist-test-slave-grzf krb5kdc[15741](info): set up 2 sockets
Sep 27 16:28:08 dist-test-slave-grzf krb5kdc[15741](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 16:28:09.191668 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:09.310108 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34151
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:09.416105 15754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:09.416349 15754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:09.416389 15754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:09.420456 15754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:09.420506 15754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:09.420524 15754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:09.420542 15754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:09.420599 15754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:09.425659 15754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:09.426626 15754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:09.427632 15754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:09.436185 15759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:09.436190 15762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:09.436266 15754 server_base.cc:1034] running on GCE node
W20260927 16:28:09.436190 15760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:09.436985 15754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:09.438054 15754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:09.439224 15754 hybrid_clock.cc:648] HybridClock initialized: now 1790526489439204 us; error 31 us; skew 500 ppm
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:09.442034 15754 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:09.442890 15754 webserver.cc:460] Webserver started at http://127.15.93.190:46397/ using document root <none> and password file <none>
I20260927 16:28:09.443506 15754 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:09.443571 15754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:09.443801 15754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:09.445829 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "799ce922fe6c4592aeac358b411e2477"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "99896276fa9766f30406e537f0954987"
server_key_iv: "79ddba29ff9b7d6c2507be03fa0ae4cb"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.446388 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "799ce922fe6c4592aeac358b411e2477"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "99896276fa9766f30406e537f0954987"
server_key_iv: "79ddba29ff9b7d6c2507be03fa0ae4cb"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.449926 15754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:09.452239 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.453378 15754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:09.453495 15754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "799ce922fe6c4592aeac358b411e2477"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "99896276fa9766f30406e537f0954987"
server_key_iv: "79ddba29ff9b7d6c2507be03fa0ae4cb"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.453591 15754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:09.472167 15754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:09.474803 15754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:09.475000 15754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:09.481628 15754 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:34151
I20260927 16:28:09.481664 15821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:34151 every 8 connection(s)
I20260927 16:28:09.482496 15754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:09.485015 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:09.485737 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 15754
I20260927 16:28:09.485877 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:09.487069 15734 external_mini_cluster.cc:1428] Setting key b3a3485cd0bd4cd92e2ccf1ddabf63ad
I20260927 16:28:09.490805 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: Bootstrap starting.
I20260927 16:28:09.492916 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:09.493592 15822 log.cc:824] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:09.495622 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: No bootstrap required, opened a new log
I20260927 16:28:09.499466 15822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } }
I20260927 16:28:09.499656 15822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:09.499712 15822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799ce922fe6c4592aeac358b411e2477, State: Initialized, Role: FOLLOWER
I20260927 16:28:09.500124 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } }
I20260927 16:28:09.500278 15822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:09.500334 15822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:09.500420 15822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:09.501367 15822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } }
I20260927 16:28:09.501684 15822 leader_election.cc:302] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 799ce922fe6c4592aeac358b411e2477; no voters:
I20260927 16:28:09.501953 15822 leader_election.cc:288] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:09.502053 15827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:09.502266 15827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [term 1 LEADER]: Becoming Leader. State: Replica: 799ce922fe6c4592aeac358b411e2477, State: Running, Role: LEADER
I20260927 16:28:09.502579 15827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } }
I20260927 16:28:09.503082 15822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:09.504233 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 799ce922fe6c4592aeac358b411e2477. Latest consensus state: current_term: 1 leader_uuid: "799ce922fe6c4592aeac358b411e2477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } } }
I20260927 16:28:09.504354 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [sys.catalog]: This master's current role is: LEADER
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:09.504710 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "799ce922fe6c4592aeac358b411e2477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "799ce922fe6c4592aeac358b411e2477" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34151 } } }
I20260927 16:28:09.504814 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477 [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:09.507963 15842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:09.508039 15842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:09.508097 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:09.508898 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:09.510828 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:09.489876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.15.93.190:34151)
0927 16:28:09.490252 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:09.490274 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:09.491345 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 16:28:09.494490 (+ 3145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:09.494509 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:09.494889 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:09.495212 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:09.497227 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.498792 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.500127 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.500810 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.500964 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:09.505753 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:09.505761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:09.505778 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:09.505802 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:09.507556 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.508050 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.508058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.508064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.508224 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.508557 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.508560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.508562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.508907 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:09.509112 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:09.509720 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 16:28:09.509965 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260927 16:28:09.514683 15836 catalog_manager.cc:1347] Generated new cluster ID: a6a092a9cc7447e591130a009e884b90
I20260927 16:28:09.514755 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:09.522655 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:09.523502 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:09.532384 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 799ce922fe6c4592aeac358b411e2477: Generated new TSK 0
I20260927 16:28:09.533051 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:09.566192 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:09.673151 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:09.673373 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:09.673461 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:09.677390 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:09.677438 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:09.677528 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:09.682484 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:09.683333 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:09.684211 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:09.692907 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:09.692930 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:09.692931 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:09.693117 15848 server_base.cc:1034] running on GCE node
I20260927 16:28:09.693552 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:09.694108 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:09.695269 15848 hybrid_clock.cc:648] HybridClock initialized: now 1790526489695256 us; error 23 us; skew 500 ppm
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:09.697811 15848 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:09.698580 15848 webserver.cc:460] Webserver started at http://127.15.93.129:38595/ using document root <none> and password file <none>
I20260927 16:28:09.699090 15848 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:09.699152 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:09.699440 15848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:09.701342 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "c7bf0382eccf461ca759caa1f66d4ff1"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "96e1baa43cddacdd3dfd451b9298112d"
server_key_iv: "6a0eea87f38d56b37dd80a1a157d6ef3"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.701881 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "c7bf0382eccf461ca759caa1f66d4ff1"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "96e1baa43cddacdd3dfd451b9298112d"
server_key_iv: "6a0eea87f38d56b37dd80a1a157d6ef3"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.705600 15848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:09.708756 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.709973 15848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:09.710067 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "c7bf0382eccf461ca759caa1f66d4ff1"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "96e1baa43cddacdd3dfd451b9298112d"
server_key_iv: "6a0eea87f38d56b37dd80a1a157d6ef3"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.710146 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:09.733031 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:09.735703 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:09.735843 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:09.736323 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:09.737043 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:09.737089 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.737126 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:09.737149 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.746006 15848 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:44027
I20260927 16:28:09.746035 15976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:44027 every 8 connection(s)
I20260927 16:28:09.746819 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:09.751770 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 15848
I20260927 16:28:09.751873 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:09.752146 15734 external_mini_cluster.cc:1428] Setting key bccb908e16f786f717d76f31b8b23b07
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:09.760748 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:09.748493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:43437 (local address 127.15.93.190:34151)
0927 16:28:09.748614 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:09.748617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:09.749392 (+ 775us) server_negotiation.cc:407] Connection header received
0927 16:28:09.750579 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:09.750582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:09.750625 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:09.750701 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:09.751800 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.752558 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.753775 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.754172 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.754287 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:09.757181 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:09.757183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:09.757185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:09.757207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:09.758589 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.759217 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.759219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.759220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.759287 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.759701 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.759703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.759704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.759866 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:09.759941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:09.760532 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 16:28:09.760624 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:28:09.761873 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:09.748839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34151 (local address 127.15.93.129:43437)
0927 16:28:09.749244 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:09.749283 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:09.750397 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:09.750819 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:09.750825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:09.751150 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:09.751655 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:09.751668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.752685 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.752690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:09.753652 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:09.753662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.754313 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.754316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:09.754445 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:09.755077 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:09.755102 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:09.757025 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:09.758710 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:09.758718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:09.758735 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:09.759118 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:09.759395 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:09.759401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:09.759407 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:09.759590 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:09.759971 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:09.759979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:09.760276 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 16:28:09.760489 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:28:09.760829 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:09.763041 15977 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34151
I20260927 16:28:09.763275 15977 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:09.763844 15977 heartbeater.cc:511] Master 127.15.93.190:34151 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:09.765359 15786 ts_manager.cc:194] Registered new tserver with Master: c7bf0382eccf461ca759caa1f66d4ff1 (127.15.93.129:44027)
I20260927 16:28:09.766489 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:43437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:09.778177 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:09.885226 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:09.885469 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:09.885512 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:09.889468 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:09.889520 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:09.889578 15982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:09.894651 15982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:09.895663 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:09.896590 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:09.905856 15987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:09.905869 15988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:09.905869 15990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:09.905920 15982 server_base.cc:1034] running on GCE node
I20260927 16:28:09.906508 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:09.907065 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:09.908226 15982 hybrid_clock.cc:648] HybridClock initialized: now 1790526489908216 us; error 25 us; skew 500 ppm
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:09.910844 15982 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:09.911713 15982 webserver.cc:460] Webserver started at http://127.15.93.130:40217/ using document root <none> and password file <none>
I20260927 16:28:09.912241 15982 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:09.912300 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:09.912542 15982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:09.914582 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "90526a12935541fb8d65fbceb173e5ad"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "d59bf5bc8d27068b76fdb92d5939f0c4"
server_key_iv: "fb05db1259330fb32fb081a67cf66421"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.915125 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "90526a12935541fb8d65fbceb173e5ad"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "d59bf5bc8d27068b76fdb92d5939f0c4"
server_key_iv: "fb05db1259330fb32fb081a67cf66421"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.918622 15982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:09.920918 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.922084 15982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:09.922212 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "90526a12935541fb8d65fbceb173e5ad"
format_stamp: "Formatted at 2026-09-27 16:28:09 on dist-test-slave-grzf"
server_key: "d59bf5bc8d27068b76fdb92d5939f0c4"
server_key_iv: "fb05db1259330fb32fb081a67cf66421"
server_key_version: "encryptionkey@0"
I20260927 16:28:09.922314 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:09.935798 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:09.938342 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:09.938527 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:09.939085 15982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:09.939864 15982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:09.939939 15982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.940006 15982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:09.940053 15982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:09.948524 15982 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:43035
I20260927 16:28:09.948547 16110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:43035 every 8 connection(s)
I20260927 16:28:09.949354 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:09.953642 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 15982
I20260927 16:28:09.953737 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:09.953990 15734 external_mini_cluster.cc:1428] Setting key ffb1df96a70d2ca15cd793077313daee
Sep 27 16:28:09 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526489, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:09.964486 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:09.951043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:40587 (local address 127.15.93.190:34151)
0927 16:28:09.951185 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:09.951189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:09.952022 (+ 833us) server_negotiation.cc:407] Connection header received
0927 16:28:09.953296 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:09.953299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:09.953357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:09.953482 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:09.954828 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.955502 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.956674 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.957030 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.957153 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:09.960666 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:09.960669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:09.960671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:09.960693 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:09.962092 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.962700 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.962702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.962703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.962771 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:09.963143 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:09.963145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:09.963146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:09.963302 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:09.963414 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:09.964163 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 16:28:09.964290 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:28:09.965700 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:09.951419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34151 (local address 127.15.93.130:40587)
0927 16:28:09.951892 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:09.951932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:09.953088 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:09.953645 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:09.953652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:09.954111 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:09.954698 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:09.954715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.955639 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.955643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:09.956553 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:09.956563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:09.957232 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:09.957239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:09.957478 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:09.958726 (+ 1248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:09.958741 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:09.960508 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:09.962213 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:09.962219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:09.962233 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:09.962590 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:09.962858 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:09.962861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:09.962863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:09.963041 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:09.963436 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:09.963445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:09.963784 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 16:28:09.964624 (+ 840us) client_negotiation.cc:240] Negotiation successful
0927 16:28:09.964886 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:09.966832 16111 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34151
I20260927 16:28:09.967072 16111 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:09.967614 16111 heartbeater.cc:511] Master 127.15.93.190:34151 requested a full tablet report, sending...
I20260927 16:28:09.968593 15786 ts_manager.cc:194] Registered new tserver with Master: 90526a12935541fb8d65fbceb173e5ad (127.15.93.130:43035)
I20260927 16:28:09.969291 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:40587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:09.980377 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:10.087314 16116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:10.087563 16116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:10.087605 16116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:10.091468 16116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:10.091513 16116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:10.091568 16116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:10.096458 16116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34151
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:10.097314 16116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:10.098192 16116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:10.107172 16122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:10.107177 16121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:10.107409 16116 server_base.cc:1034] running on GCE node
W20260927 16:28:10.107177 16124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:10.107817 16116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:10.108386 16116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:10.109537 16116 hybrid_clock.cc:648] HybridClock initialized: now 1790526490109523 us; error 27 us; skew 500 ppm
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:10.112087 16116 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:10.112814 16116 webserver.cc:460] Webserver started at http://127.15.93.131:45157/ using document root <none> and password file <none>
I20260927 16:28:10.113410 16116 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:10.113472 16116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:10.113631 16116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:10.115617 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "d07e674066f64caa99e0916c7a99c6b3"
format_stamp: "Formatted at 2026-09-27 16:28:10 on dist-test-slave-grzf"
server_key: "c0f51e5093acc52be32240e3e7cc2f56"
server_key_iv: "585d3e099f7c6227a99fc1a146188e03"
server_key_version: "encryptionkey@0"
I20260927 16:28:10.116120 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "d07e674066f64caa99e0916c7a99c6b3"
format_stamp: "Formatted at 2026-09-27 16:28:10 on dist-test-slave-grzf"
server_key: "c0f51e5093acc52be32240e3e7cc2f56"
server_key_iv: "585d3e099f7c6227a99fc1a146188e03"
server_key_version: "encryptionkey@0"
I20260927 16:28:10.119553 16116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:10.121783 16131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:10.122897 16116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:10.123010 16116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "d07e674066f64caa99e0916c7a99c6b3"
format_stamp: "Formatted at 2026-09-27 16:28:10 on dist-test-slave-grzf"
server_key: "c0f51e5093acc52be32240e3e7cc2f56"
server_key_iv: "585d3e099f7c6227a99fc1a146188e03"
server_key_version: "encryptionkey@0"
I20260927 16:28:10.123102 16116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:10.139197 16116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:10.142453 16116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:10.142660 16116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:10.143462 16116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:10.144563 16116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:10.144618 16116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:10.144672 16116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:10.144732 16116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 16:28:10.153751 16116 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40443
I20260927 16:28:10.153772 16244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40443 every 8 connection(s)
I20260927 16:28:10.154595 16116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:10.155793 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16116
I20260927 16:28:10.155917 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:10.156184 15734 external_mini_cluster.cc:1428] Setting key eadf347ab986ef01c9086ac9cde6057c
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:10.167584 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:10.156415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:48687 (local address 127.15.93.190:34151)
0927 16:28:10.156569 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:10.156573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:10.157995 (+ 1422us) server_negotiation.cc:407] Connection header received
0927 16:28:10.158921 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:10.158924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:10.158969 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:10.159040 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:10.160161 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.160675 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.161713 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.161913 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.161992 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:10.164199 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:10.164202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:10.164204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:10.164225 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:10.165410 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.165919 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.165924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.165929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.166026 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.166476 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.166497 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.166500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.166731 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:10.166835 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:10.167286 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 16:28:10.167421 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:10.168314 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:10.156855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34151 (local address 127.15.93.131:48687)
0927 16:28:10.157859 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:10.157908 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:10.158743 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:10.159163 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:10.159169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:10.159582 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:10.160036 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:10.160049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.160775 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.160777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:10.161608 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:10.161617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.162029 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.162031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:10.162171 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:10.162691 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:10.162706 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:10.164048 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:10.165529 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:10.165538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:10.165553 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:10.165836 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:10.166171 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:10.166174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:10.166175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:10.166370 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:10.166833 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:10.166840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:10.167062 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 16:28:10.167245 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 16:28:10.167574 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":651,"threads_started":1}
I20260927 16:28:10.169410 16245 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34151
I20260927 16:28:10.169659 16245 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:10.170166 16245 heartbeater.cc:511] Master 127.15.93.190:34151 requested a full tablet report, sending...
I20260927 16:28:10.171108 15786 ts_manager.cc:194] Registered new tserver with Master: d07e674066f64caa99e0916c7a99c6b3 (127.15.93.131:40443)
I20260927 16:28:10.171737 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:48687
I20260927 16:28:10.181659 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:10.206714 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:10.197288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.15.93.129:44027)
0927 16:28:10.197642 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:10.197655 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:10.197683 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:28:10.197765 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:10.197770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:10.197956 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:10.198102 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:10.198770 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.199598 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.200497 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.200859 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.200973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:10.203340 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:10.203349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:10.203361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:10.203428 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:10.204967 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.205407 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.205412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.205414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.205498 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.205775 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.205778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.205780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.206014 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:10.206231 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:10.206391 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:28:10.206537 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":131,"threads_started":1}
W20260927 16:28:10.207669 15891 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36818
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:10.231593 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:10.223881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.15.93.129:44027)
0927 16:28:10.224055 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:10.224059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:10.224118 (+ 59us) server_negotiation.cc:407] Connection header received
0927 16:28:10.224225 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:10.224228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:10.224267 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:10.224334 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:10.224953 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.225581 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.226371 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:10.226667 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:10.226745 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:10.229039 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:10.229042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:10.229044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:10.229065 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:10.230203 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.230543 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.230546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.230547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.230618 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:10.230840 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:10.230842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:10.230843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:10.230988 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:10.231081 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:10.231262 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:28:10.231352 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 16:28:10.232532 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 15848
I20260927 16:28:10.238077 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 15982
I20260927 16:28:10.243098 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16116
I20260927 16:28:10.248131 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 15754
2026-09-27T16:28:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2106 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[16259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[16259](info): set up 2 sockets
Sep 27 16:28:10 dist-test-slave-grzf krb5kdc[16259](info): commencing operation
krb5kdc: starting...
W20260927 16:28:11.275610 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:11 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:11.392660 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38691
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:11.502226 16272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:11.502548 16272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:11.502635 16272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:11.506687 16272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:11.506774 16272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:11.506812 16272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:11.506850 16272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:11.506886 16272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:11.512058 16272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:11.513069 16272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:11.514072 16272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:11.522058 16278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:11.522058 16277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:11.522044 16280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:11.522130 16272 server_base.cc:1034] running on GCE node
I20260927 16:28:11.522836 16272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:11.523962 16272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:11.525120 16272 hybrid_clock.cc:648] HybridClock initialized: now 1790526491525088 us; error 42 us; skew 500 ppm
Sep 27 16:28:11 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:11.527791 16272 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:11.528642 16272 webserver.cc:460] Webserver started at http://127.15.93.190:40925/ using document root <none> and password file <none>
I20260927 16:28:11.529175 16272 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:11.529263 16272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:11.529479 16272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:11.531462 16272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "62ee9638401b4e47a19a0489d5a16a86"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "edfd9ca21bda8fac8ced1b874a980f95"
server_key_iv: "908b1fd4cc24392356edb1468fc29038"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.532009 16272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "62ee9638401b4e47a19a0489d5a16a86"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "edfd9ca21bda8fac8ced1b874a980f95"
server_key_iv: "908b1fd4cc24392356edb1468fc29038"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.535705 16272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 16:28:11.537942 16287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:11.539177 16272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:11.539310 16272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "62ee9638401b4e47a19a0489d5a16a86"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "edfd9ca21bda8fac8ced1b874a980f95"
server_key_iv: "908b1fd4cc24392356edb1468fc29038"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.539466 16272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:11.562683 16272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:11.565659 16272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:11.565860 16272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:11.573151 16272 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38691
I20260927 16:28:11.573179 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38691 every 8 connection(s)
I20260927 16:28:11.574064 16272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:11.576766 16340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:11.579021 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16272
I20260927 16:28:11.579107 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:11.579363 15734 external_mini_cluster.cc:1428] Setting key c7d7b68831f0a586a6c731ad60b225bf
I20260927 16:28:11.582808 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: Bootstrap starting.
I20260927 16:28:11.585368 16340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:11.586180 16340 log.cc:824] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:11.588063 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: No bootstrap required, opened a new log
Sep 27 16:28:11 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:11.592092 16340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } }
I20260927 16:28:11.592383 16340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:11.592458 16340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ee9638401b4e47a19a0489d5a16a86, State: Initialized, Role: FOLLOWER
I20260927 16:28:11.592955 16340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } }
I20260927 16:28:11.593078 16340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:11.593154 16340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:11.593250 16340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:11.594216 16340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } }
I20260927 16:28:11.594527 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:11.580734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.15.93.190:38691)
0927 16:28:11.581148 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:11.581170 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:11.581213 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:28:11.582028 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:11.582050 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:11.582452 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:11.582804 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:11.583512 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.584990 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.585839 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.586536 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.586685 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:11.589182 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:11.589188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:11.589203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:11.589231 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:11.591788 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:11.592185 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:11.592190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:11.592196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:11.592327 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:11.592668 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:11.592671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:11.592672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:11.593041 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:11.593270 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:11.593559 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 16:28:11.593834 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260927 16:28:11.594584 16340 leader_election.cc:302] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ee9638401b4e47a19a0489d5a16a86; no voters:
I20260927 16:28:11.594945 16340 leader_election.cc:288] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:11.595011 16345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:11.595208 16345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [term 1 LEADER]: Becoming Leader. State: Replica: 62ee9638401b4e47a19a0489d5a16a86, State: Running, Role: LEADER
I20260927 16:28:11.595541 16345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } }
I20260927 16:28:11.596200 16340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:11.597584 16346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ee9638401b4e47a19a0489d5a16a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } } }
I20260927 16:28:11.597988 16346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:11.597882 16347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ee9638401b4e47a19a0489d5a16a86. Latest consensus state: current_term: 1 leader_uuid: "62ee9638401b4e47a19a0489d5a16a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ee9638401b4e47a19a0489d5a16a86" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38691 } } }
I20260927 16:28:11.598152 16347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86 [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:11.600819 16360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:11.600883 16360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:11.600961 16361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:11.601680 16361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:11.606401 16361 catalog_manager.cc:1347] Generated new cluster ID: 302dd1be746346ae9b7afe70c198e640
I20260927 16:28:11.606460 16361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:11.630065 16361 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:11.630988 16361 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:11.638198 16361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ee9638401b4e47a19a0489d5a16a86: Generated new TSK 0
I20260927 16:28:11.638875 16361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:11.668370 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:11.777837 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:11.778077 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:11.778120 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:11.782147 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:11.782200 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:11.782256 16366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:11.787328 16366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:11.788370 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:11.789371 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:11.798796 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:11.798846 16366 server_base.cc:1034] running on GCE node
W20260927 16:28:11.798770 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:11.798785 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:11.799535 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:11.800081 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:11.801242 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790526491801226 us; error 28 us; skew 500 ppm
Sep 27 16:28:11 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:11.803886 16366 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:11.804697 16366 webserver.cc:460] Webserver started at http://127.15.93.129:44147/ using document root <none> and password file <none>
I20260927 16:28:11.805236 16366 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:11.805323 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:11.805531 16366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:11.807555 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "75bf0b3ac20f466e923714f827dd1235"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "591b123a665036a6a537d9e64c25561b"
server_key_iv: "25321cd71cd6d2e4a1aa14115209e17c"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.808112 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "75bf0b3ac20f466e923714f827dd1235"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "591b123a665036a6a537d9e64c25561b"
server_key_iv: "25321cd71cd6d2e4a1aa14115209e17c"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.811600 16366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:11.813969 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:11.815166 16366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:11.815313 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "75bf0b3ac20f466e923714f827dd1235"
format_stamp: "Formatted at 2026-09-27 16:28:11 on dist-test-slave-grzf"
server_key: "591b123a665036a6a537d9e64c25561b"
server_key_iv: "25321cd71cd6d2e4a1aa14115209e17c"
server_key_version: "encryptionkey@0"
I20260927 16:28:11.815477 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:11.831459 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:11.834228 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:11.834429 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:11.835003 16366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:11.835943 16366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:11.836018 16366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:11.836086 16366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:11.836115 16366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:11.844907 16366 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:34743
I20260927 16:28:11.844924 16494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:34743 every 8 connection(s)
I20260927 16:28:11.845801 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:11.854102 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16366
I20260927 16:28:11.854195 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
Sep 27 16:28:11 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:11.854512 15734 external_mini_cluster.cc:1428] Setting key 733138104c7a1c8c8f1df3cc660f7c31
I20260927 16:28:11.858810 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:11.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:51495 (local address 127.15.93.190:38691)
0927 16:28:11.847646 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:11.847651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:11.848492 (+ 841us) server_negotiation.cc:407] Connection header received
0927 16:28:11.849304 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:11.849307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:11.849353 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:11.849425 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:11.850478 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.851106 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.852256 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.852487 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.852570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:11.854978 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:11.854982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:11.854985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:11.855012 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:11.856532 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:11.857164 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:11.857166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:11.857168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:11.857236 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:11.857616 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:11.857621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:11.857624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:11.857894 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:11.857999 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:11.858458 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 16:28:11.858605 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:11.859692 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:11.847899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38691 (local address 127.15.93.129:51495)
0927 16:28:11.848334 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:11.848377 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:11.849105 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:11.849549 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:11.849555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:11.849885 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:11.850357 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:11.850370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.851216 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.851219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:11.852144 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:11.852154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:11.852613 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:11.852616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:11.852748 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:11.853250 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:11.853265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:11.854804 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:11.856670 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:11.856679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:11.856697 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:11.857059 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:11.857320 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:11.857326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:11.857331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:11.857515 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:11.858004 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:11.858009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:11.858230 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 16:28:11.858384 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 16:28:11.858621 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260927 16:28:11.860870 16495 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38691
I20260927 16:28:11.861088 16495 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:11.861559 16495 heartbeater.cc:511] Master 127.15.93.190:38691 requested a full tablet report, sending...
I20260927 16:28:11.862963 16304 ts_manager.cc:194] Registered new tserver with Master: 75bf0b3ac20f466e923714f827dd1235 (127.15.93.129:34743)
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:11.864169 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:51495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:11.880429 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:11.987017 16500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:11.987257 16500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:11.987346 16500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:11.991312 16500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:11.991410 16500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:11.991516 16500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:11.996811 16500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:11.997840 16500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:11.998802 16500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:28:12.008000 16500 server_base.cc:1034] running on GCE node
W20260927 16:28:12.007988 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:12.007997 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:12.008096 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:12.008662 16500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:12.009269 16500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:12.010456 16500 hybrid_clock.cc:648] HybridClock initialized: now 1790526492010442 us; error 26 us; skew 500 ppm
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:12.013147 16500 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:12.013952 16500 webserver.cc:460] Webserver started at http://127.15.93.130:44811/ using document root <none> and password file <none>
I20260927 16:28:12.014495 16500 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:12.014585 16500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:12.014802 16500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:12.016827 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "4ede2949f6895dad39a147c55991a93d"
server_key_iv: "0cfc4c9c67918b90a6585b28efe86d39"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.017372 16500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "4ede2949f6895dad39a147c55991a93d"
server_key_iv: "0cfc4c9c67918b90a6585b28efe86d39"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.020920 16500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:12.023173 16515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.024402 16500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:12.024539 16500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "4ede2949f6895dad39a147c55991a93d"
server_key_iv: "0cfc4c9c67918b90a6585b28efe86d39"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.024643 16500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:12.049126 16500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:12.052454 16500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:12.052660 16500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:12.053242 16500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:12.054044 16500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:12.054111 16500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.054206 16500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:12.054257 16500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.062863 16500 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:45437
I20260927 16:28:12.062897 16628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:45437 every 8 connection(s)
I20260927 16:28:12.063745 16500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:12.065765 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16500
I20260927 16:28:12.065888 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:12.066135 15734 external_mini_cluster.cc:1428] Setting key 64f40363dca37787138b6def73bb8317
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:12.077412 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.065406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:53781 (local address 127.15.93.190:38691)
0927 16:28:12.065555 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.065560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.066567 (+ 1007us) server_negotiation.cc:407] Connection header received
0927 16:28:12.067744 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.067747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.067795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.067862 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.069105 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.069686 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.070841 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.071079 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.071190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.073680 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.073684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.073686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.073718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.075285 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.075895 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.075898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.075900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.075978 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.076319 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.076322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.076324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.076502 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.076608 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.077104 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.077240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:28:12.078223 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.065834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38691 (local address 127.15.93.130:53781)
0927 16:28:12.066378 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.066443 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.067479 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.068089 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.068097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.068453 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:12.068933 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.068945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.069800 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.069803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.070713 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.070722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.071227 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.071230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.071482 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.072104 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:12.072120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:12.073508 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:12.075420 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:12.075426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:12.075440 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:12.075786 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:12.076075 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:12.076078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:12.076080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:12.076219 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:12.076606 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:12.076613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:12.076869 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.077049 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.077371 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260927 16:28:12.079612 16629 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38691
I20260927 16:28:12.079874 16629 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:12.080448 16629 heartbeater.cc:511] Master 127.15.93.190:38691 requested a full tablet report, sending...
I20260927 16:28:12.081581 16304 ts_manager.cc:194] Registered new tserver with Master: b1ea0a0d23f54737ba7308b5e4d55f0b (127.15.93.130:45437)
I20260927 16:28:12.082460 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:53781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:12.093451 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:12.213850 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:12.214089 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:12.214133 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:12.218230 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:12.218290 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:12.218348 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:12.223914 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:12.224874 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:12.225802 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:12.235140 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:12.235122 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:12.235208 16634 server_base.cc:1034] running on GCE node
W20260927 16:28:12.235155 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:12.235698 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:12.236403 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:12.237567 16634 hybrid_clock.cc:648] HybridClock initialized: now 1790526492237537 us; error 39 us; skew 500 ppm
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:12.240296 16634 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:12.241042 16634 webserver.cc:460] Webserver started at http://127.15.93.131:34167/ using document root <none> and password file <none>
I20260927 16:28:12.241552 16634 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:12.241607 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:12.241822 16634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:12.243947 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "30320dd113754f5ca1a0e2ad3abc60a8"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "11ff8d9bff20cfe25a61b68a813b43d6"
server_key_iv: "b6275eb02a48d93f73709845e4bbe6aa"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.244472 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "30320dd113754f5ca1a0e2ad3abc60a8"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "11ff8d9bff20cfe25a61b68a813b43d6"
server_key_iv: "b6275eb02a48d93f73709845e4bbe6aa"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.247923 16634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:12.250171 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.251296 16634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:12.251441 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "30320dd113754f5ca1a0e2ad3abc60a8"
format_stamp: "Formatted at 2026-09-27 16:28:12 on dist-test-slave-grzf"
server_key: "11ff8d9bff20cfe25a61b68a813b43d6"
server_key_iv: "b6275eb02a48d93f73709845e4bbe6aa"
server_key_version: "encryptionkey@0"
I20260927 16:28:12.251535 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:12.267067 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:12.270272 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:12.270462 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:12.271024 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:12.271905 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:12.271950 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.272013 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:12.272046 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:12.280280 16634 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:33103
I20260927 16:28:12.280305 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:33103 every 8 connection(s)
I20260927 16:28:12.281113 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:12.290119 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16634
I20260927 16:28:12.290251 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:12.290560 15734 external_mini_cluster.cc:1428] Setting key 3bd5a7b1d50ae5c8704b9ca0ab1169fc
I20260927 16:28:12.293941 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.282949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:46513 (local address 127.15.93.190:38691)
0927 16:28:12.283071 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.283075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.283886 (+ 811us) server_negotiation.cc:407] Connection header received
0927 16:28:12.284747 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.284750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.284809 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.284885 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.285972 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.286504 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.287695 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.287926 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.288032 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.290396 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.290400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.290405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.290432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.292004 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.292489 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.292492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.292494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.292580 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.292925 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.292928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.292930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.293112 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.293216 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.293676 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.293786 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:28:12.294734 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.283279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38691 (local address 127.15.93.131:46513)
0927 16:28:12.283731 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.283771 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.284541 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.285042 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.285049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.285377 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:12.285827 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.285840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.286644 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.286647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.287566 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.287575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.288066 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.288071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.288232 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.288795 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:12.288810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:12.290154 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:12.292116 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:12.292122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:12.292135 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:12.292384 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:12.292686 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:12.292689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:12.292691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:12.292822 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:12.293214 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:12.293221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:12.293447 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.293635 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.293932 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260927 16:28:12.295782 16763 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38691
I20260927 16:28:12.296032 16763 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:12.296509 16763 heartbeater.cc:511] Master 127.15.93.190:38691 requested a full tablet report, sending...
I20260927 16:28:12.297557 16304 ts_manager.cc:194] Registered new tserver with Master: 30320dd113754f5ca1a0e2ad3abc60a8 (127.15.93.131:33103)
I20260927 16:28:12.298063 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:46513
I20260927 16:28:12.304201 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:28:12.314155 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.307363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.15.93.190:38691)
0927 16:28:12.307514 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.307517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.307633 (+ 116us) server_negotiation.cc:407] Connection header received
0927 16:28:12.307720 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.307723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.307762 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.307830 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.308447 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.308939 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.309696 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.309893 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.309967 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.311276 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.311279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.311281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.311302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.312764 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.313121 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.313124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.313125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.313195 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.313449 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.313452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.313453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.313687 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.313791 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.313959 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.314041 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 16:28:12.318691 16304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:12.321753 16304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:12.335790 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.330119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:34743 (local address 127.0.0.1:33848)
0927 16:28:12.330516 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.330570 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.330694 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.331536 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.331542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.331567 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:12.331736 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.331743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333124 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.333138 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.334073 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.334082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.335218 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.335223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.335337 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.335404 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.335527 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.335641 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":115,"threads_started":1}
I20260927 16:28:12.335820 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.330297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.15.93.129:34743)
0927 16:28:12.330836 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.330853 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.330877 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:28:12.330962 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.330970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.331189 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.331901 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:12.332055 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333002 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.334217 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.335042 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.335203 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.335363 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.335574 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.335675 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":104,"threads_started":1}
I20260927 16:28:12.336663 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.330861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45437 (local address 127.0.0.1:43546)
0927 16:28:12.331459 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.331471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.331564 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.332264 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.332266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.332281 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:12.332488 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.332499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333922 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.333925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.335117 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.335129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.336281 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.336285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.336374 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.336396 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.336462 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.336554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":62,"thread_start_us":111,"threads_started":1}
I20260927 16:28:12.337227 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.330510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:33103 (local address 127.0.0.1:51042)
0927 16:28:12.331131 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.331145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.331227 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.332234 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.332240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.332262 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:12.332463 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.332474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333956 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.333967 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.335003 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.335013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.336826 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.336829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.336916 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.336937 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.337038 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.337104 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":61,"spinlock_wait_cycles":17408,"thread_start_us":88,"threads_started":1}
I20260927 16:28:12.337513 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.331287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.15.93.130:45437)
0927 16:28:12.331811 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.331825 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.331848 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:28:12.331925 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.331931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.332112 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.332421 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:12.332605 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333772 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.335246 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.336111 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.336261 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.336544 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.336591 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.336678 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":157,"threads_started":1}
I20260927 16:28:12.337860 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.330921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.15.93.131:33103)
0927 16:28:12.331773 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.331787 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.331814 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:28:12.331889 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.331894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.332075 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.332240 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:12.332606 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.333772 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.335126 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.336159 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.336297 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.336433 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.337069 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.337709 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":174,"threads_started":1}
I20260927 16:28:12.338991 16429 tablet_service.cc:1467] Processing CreateTablet for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1 (DEFAULT_TABLE table=test-table [id=071138c9c4944a7295c332547ba6874c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:12.339982 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:12.340368 16697 tablet_service.cc:1467] Processing CreateTablet for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1 (DEFAULT_TABLE table=test-table [id=071138c9c4944a7295c332547ba6874c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:12.340368 16563 tablet_service.cc:1467] Processing CreateTablet for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1 (DEFAULT_TABLE table=test-table [id=071138c9c4944a7295c332547ba6874c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:12.341382 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:12.341387 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:12.345870 16780 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Bootstrap starting.
I20260927 16:28:12.348198 16780 tablet_bootstrap.cc:654] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:12.348491 16782 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Bootstrap starting.
I20260927 16:28:12.349188 16781 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Bootstrap starting.
I20260927 16:28:12.349581 16780 log.cc:824] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:12.350589 16782 tablet_bootstrap.cc:654] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:12.351420 16781 tablet_bootstrap.cc:654] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:12.351760 16780 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: No bootstrap required, opened a new log
I20260927 16:28:12.351780 16782 log.cc:824] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:12.351951 16780 ts_tablet_manager.cc:1397] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:28:12.352201 16781 log.cc:824] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:12.353752 16781 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: No bootstrap required, opened a new log
I20260927 16:28:12.353961 16781 ts_tablet_manager.cc:1397] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:12.354493 16782 tablet_bootstrap.cc:492] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: No bootstrap required, opened a new log
I20260927 16:28:12.354709 16782 ts_tablet_manager.cc:1397] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:28:12.355940 16780 raft_consensus.cc:348] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.356206 16780 raft_consensus.cc:374] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:12.356261 16780 raft_consensus.cc:729] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bf0b3ac20f466e923714f827dd1235, State: Initialized, Role: FOLLOWER
I20260927 16:28:12.356710 16780 consensus_queue.cc:260] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.357358 16495 heartbeater.cc:503] Master 127.15.93.190:38691 was elected leader, sending a full tablet report...
I20260927 16:28:12.357434 16781 raft_consensus.cc:348] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.357681 16781 raft_consensus.cc:374] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:12.357733 16781 raft_consensus.cc:729] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ea0a0d23f54737ba7308b5e4d55f0b, State: Initialized, Role: FOLLOWER
I20260927 16:28:12.357738 16780 ts_tablet_manager.cc:1428] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 16:28:12.358104 16781 consensus_queue.cc:260] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.358385 16782 raft_consensus.cc:348] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.358637 16782 raft_consensus.cc:374] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:12.358688 16782 raft_consensus.cc:729] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30320dd113754f5ca1a0e2ad3abc60a8, State: Initialized, Role: FOLLOWER
I20260927 16:28:12.358700 16629 heartbeater.cc:503] Master 127.15.93.190:38691 was elected leader, sending a full tablet report...
I20260927 16:28:12.358999 16781 ts_tablet_manager.cc:1428] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:28:12.359036 16782 consensus_queue.cc:260] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.359721 16763 heartbeater.cc:503] Master 127.15.93.190:38691 was elected leader, sending a full tablet report...
I20260927 16:28:12.361060 16782 ts_tablet_manager.cc:1428] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 16:28:12.532099 16764 tablet.cc:2369] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:28:12.564808 16630 tablet.cc:2369] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:28:12.597215 16496 tablet.cc:2369] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:12.620247 16788 raft_consensus.cc:482] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:12.620507 16788 raft_consensus.cc:504] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.621654 16788 leader_election.cc:288] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bf0b3ac20f466e923714f827dd1235 (127.15.93.129:34743), b1ea0a0d23f54737ba7308b5e4d55f0b (127.15.93.130:45437)
I20260927 16:28:12.625331 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.621922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:34743 (local address 127.15.93.131:38983)
0927 16:28:12.622060 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.622076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.622180 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.622518 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.622521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.622538 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:12.622668 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.622674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.623418 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.623421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.624192 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.624200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.624908 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.624912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.625020 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.625047 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.625132 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.625200 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260927 16:28:12.625330 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.621994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38983 (local address 127.15.93.129:34743)
0927 16:28:12.622128 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.622132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.622147 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:12.622339 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.622342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.622403 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.622500 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:12.622778 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.623282 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.624311 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.624793 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.624921 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.624974 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.625155 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.625221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:28:12.625800 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.622161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45437 (local address 127.15.93.131:56499)
0927 16:28:12.622545 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:12.622555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:12.622665 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:12.622922 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:12.622925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:12.622940 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:12.623071 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.623076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.623831 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.623836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.624730 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:12.624740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.625373 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.625376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:12.625500 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.625533 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 16:28:12.625607 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 16:28:12.625670 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260927 16:28:12.625790 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.622323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:56499 (local address 127.15.93.130:45437)
0927 16:28:12.622472 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.622476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.622628 (+ 152us) server_negotiation.cc:407] Connection header received
0927 16:28:12.622768 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.622771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.622820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.622912 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:12.623174 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.623726 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.624849 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.625242 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.625372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.625418 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.625623 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.625689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:28:12.625967 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e666f3bfbbc47c19cec9d4a286b1ff1" candidate_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf0b3ac20f466e923714f827dd1235" is_pre_election: true
I20260927 16:28:12.626295 16449 raft_consensus.cc:2454] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30320dd113754f5ca1a0e2ad3abc60a8 in term 0.
I20260927 16:28:12.626629 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e666f3bfbbc47c19cec9d4a286b1ff1" candidate_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" is_pre_election: true
I20260927 16:28:12.626813 16652 leader_election.cc:302] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30320dd113754f5ca1a0e2ad3abc60a8, 75bf0b3ac20f466e923714f827dd1235; no voters:
I20260927 16:28:12.626925 16583 raft_consensus.cc:2454] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30320dd113754f5ca1a0e2ad3abc60a8 in term 0.
I20260927 16:28:12.627100 16788 raft_consensus.cc:2781] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:12.627192 16788 raft_consensus.cc:482] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:12.627252 16788 raft_consensus.cc:3037] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:12.628331 16788 raft_consensus.cc:504] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.628666 16788 leader_election.cc:288] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [CANDIDATE]: Term 1 election: Requested vote from peers 75bf0b3ac20f466e923714f827dd1235 (127.15.93.129:34743), b1ea0a0d23f54737ba7308b5e4d55f0b (127.15.93.130:45437)
I20260927 16:28:12.629077 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e666f3bfbbc47c19cec9d4a286b1ff1" candidate_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf0b3ac20f466e923714f827dd1235"
I20260927 16:28:12.629077 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e666f3bfbbc47c19cec9d4a286b1ff1" candidate_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b"
I20260927 16:28:12.629202 16583 raft_consensus.cc:3037] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:12.629202 16449 raft_consensus.cc:3037] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:12.630494 16449 raft_consensus.cc:2454] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 75bf0b3ac20f466e923714f827dd1235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30320dd113754f5ca1a0e2ad3abc60a8 in term 1.
I20260927 16:28:12.630494 16583 raft_consensus.cc:2454] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P b1ea0a0d23f54737ba7308b5e4d55f0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30320dd113754f5ca1a0e2ad3abc60a8 in term 1.
I20260927 16:28:12.630805 16652 leader_election.cc:302] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30320dd113754f5ca1a0e2ad3abc60a8, 75bf0b3ac20f466e923714f827dd1235; no voters:
I20260927 16:28:12.630961 16788 raft_consensus.cc:2781] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:12.631189 16788 raft_consensus.cc:686] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [term 1 LEADER]: Becoming Leader. State: Replica: 30320dd113754f5ca1a0e2ad3abc60a8, State: Running, Role: LEADER
I20260927 16:28:12.631561 16788 consensus_queue.cc:237] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } }
I20260927 16:28:12.634586 16303 catalog_manager.cc:5495] T 0e666f3bfbbc47c19cec9d4a286b1ff1 P 30320dd113754f5ca1a0e2ad3abc60a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30320dd113754f5ca1a0e2ad3abc60a8 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf0b3ac20f466e923714f827dd1235" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ea0a0d23f54737ba7308b5e4d55f0b" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30320dd113754f5ca1a0e2ad3abc60a8" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33103 } health_report { overall_health: HEALTHY } } }
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:12.700263 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.691904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.15.93.129:34743)
0927 16:28:12.692033 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.692037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.692118 (+ 81us) server_negotiation.cc:407] Connection header received
0927 16:28:12.692256 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.692258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.692299 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.692367 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.693019 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.693505 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.694263 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.694462 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.694548 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.696923 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.696934 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.696949 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.696972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.698711 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.699075 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.699079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.699081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.699161 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.699415 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.699423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.699425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.699669 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.699827 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.699955 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.700047 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:12.726473 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.717615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.15.93.129:34743)
0927 16:28:12.717777 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.717784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.718028 (+ 244us) server_negotiation.cc:407] Connection header received
0927 16:28:12.718268 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.718272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.718327 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.718416 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.719292 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.720020 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.720864 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.721105 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.721215 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.723508 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.723512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.723514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.723546 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.724833 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.725238 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.725243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.725245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.725329 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.725594 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.725598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.725600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.725771 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.725906 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.726186 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.726308 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:12.751027 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:12.742831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.15.93.129:34743)
0927 16:28:12.742962 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:12.742967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:12.743131 (+ 164us) server_negotiation.cc:407] Connection header received
0927 16:28:12.743319 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:12.743322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:12.743364 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:12.743509 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:12.744260 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.744765 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.745587 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:12.745803 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:12.745903 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:12.748263 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:12.748267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:12.748269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:12.748295 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:12.749473 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.749879 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.749884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.749886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.749966 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:12.750232 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:12.750237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:12.750238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:12.750397 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:12.750524 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:12.750745 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:28:12.750858 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:28:12.752323 16392 tablet_service.cc:2908] Rejecting scan request for tablet 0e666f3bfbbc47c19cec9d4a286b1ff1: Uninitialized: clean time has not yet been initialized
I20260927 16:28:12.754413 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16366
I20260927 16:28:12.760277 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16500
I20260927 16:28:12.765657 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16634
I20260927 16:28:12.770882 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16272
2026-09-27T16:28:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2522 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16804](info): set up 2 sockets
Sep 27 16:28:12 dist-test-slave-grzf krb5kdc[16804](info): commencing operation
krb5kdc: starting...
W20260927 16:28:13.799492 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:13 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:13.913532 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42253
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:14.018783 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:14.019062 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:14.019135 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:14.023007 16817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:14.023090 16817 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:28:14.023120 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:14.023173 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:14.023233 16817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:14.023272 16817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:14.028353 16817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:42253
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:14.029297 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:14.030234 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:14.038081 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.038075 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.038085 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:14.038322 16817 server_base.cc:1034] running on GCE node
I20260927 16:28:14.038946 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:14.039973 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:14.041126 16817 hybrid_clock.cc:648] HybridClock initialized: now 1790526494041100 us; error 37 us; skew 500 ppm
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:14.048328 16817 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:14.049175 16817 webserver.cc:460] Webserver started at http://127.15.93.190:36721/ using document root <none> and password file <none>
I20260927 16:28:14.049722 16817 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:14.049780 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:14.050024 16817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:14.052069 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "b2d535f9aff44c008662088863a6a39e"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3aacda8c602419cb12a1ef860bb52265"
server_key_iv: "21d864157c270111c7f456210932f998"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.052616 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "b2d535f9aff44c008662088863a6a39e"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3aacda8c602419cb12a1ef860bb52265"
server_key_iv: "21d864157c270111c7f456210932f998"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.056154 16817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:14.058327 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.059538 16817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:14.059679 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "b2d535f9aff44c008662088863a6a39e"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3aacda8c602419cb12a1ef860bb52265"
server_key_iv: "21d864157c270111c7f456210932f998"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.059782 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:14.077754 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:14.080675 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:14.080864 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:14.088095 16817 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:42253
I20260927 16:28:14.088115 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:42253 every 8 connection(s)
I20260927 16:28:14.089023 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:14.091622 16895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:14.096767 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Bootstrap starting.
I20260927 16:28:14.098881 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16817
I20260927 16:28:14.098970 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:14.099059 16895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:14.099277 15734 external_mini_cluster.cc:1428] Setting key 1086f0a64a0e33e1388bc5ac219f084f
I20260927 16:28:14.099931 16895 log.cc:824] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:14.101763 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: No bootstrap required, opened a new log
I20260927 16:28:14.105875 16895 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } }
I20260927 16:28:14.106145 16895 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:14.106213 16895 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d535f9aff44c008662088863a6a39e, State: Initialized, Role: FOLLOWER
I20260927 16:28:14.106693 16895 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } }
I20260927 16:28:14.106873 16895 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:14.106942 16895 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:14.107034 16895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:14.107964 16895 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } }
I20260927 16:28:14.108279 16895 leader_election.cc:302] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d535f9aff44c008662088863a6a39e; no voters:
I20260927 16:28:14.108558 16895 leader_election.cc:288] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:14.108691 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:14.108934 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [term 1 LEADER]: Becoming Leader. State: Replica: b2d535f9aff44c008662088863a6a39e, State: Running, Role: LEADER
I20260927 16:28:14.109308 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } }
I20260927 16:28:14.109802 16895 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:14.111152 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d535f9aff44c008662088863a6a39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } } }
I20260927 16:28:14.111277 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:14.111521 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d535f9aff44c008662088863a6a39e. Latest consensus state: current_term: 1 leader_uuid: "b2d535f9aff44c008662088863a6a39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d535f9aff44c008662088863a6a39e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42253 } } }
I20260927 16:28:14.111642 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:14.113829 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.100310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.15.93.190:42253)
0927 16:28:14.100753 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.100777 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.100810 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:28:14.101393 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.101408 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.101704 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.102015 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.102822 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.103867 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.105032 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.105484 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.105592 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.108384 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.108389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.108420 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.108452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.110647 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.111136 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.111144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.111149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.111289 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.111640 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.111645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.111647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.112155 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.112351 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.112606 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.112901 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
W20260927 16:28:14.114547 16915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:14.114637 16915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:14.114733 16914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:14.115597 16914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:14.120354 16914 catalog_manager.cc:1347] Generated new cluster ID: 2cd695dde5fa4b21a0319864be7dc4d0
I20260927 16:28:14.120411 16914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:14.154989 16914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:14.156299 16914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:14.171861 16914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Generated new TSK 0
I20260927 16:28:14.172658 16914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:14.207890 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:42253
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:14.315515 16921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:14.315788 16921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:14.315861 16921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:14.319768 16921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:14.319841 16921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:14.319909 16921 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:14.319993 16921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:14.324980 16921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:42253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:14.325934 16921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:14.326890 16921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:14.336053 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.336016 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.336014 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:14.336355 16921 server_base.cc:1034] running on GCE node
I20260927 16:28:14.336724 16921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:14.337270 16921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:14.338424 16921 hybrid_clock.cc:648] HybridClock initialized: now 1790526494338394 us; error 42 us; skew 500 ppm
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:14.341253 16921 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:14.342074 16921 webserver.cc:460] Webserver started at http://127.15.93.129:33359/ using document root <none> and password file <none>
I20260927 16:28:14.342617 16921 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:14.342702 16921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:14.342904 16921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:14.345121 16921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "f9f2fe51b09140acbf1ed42d1c6452cc"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "4b06d754418e58eb8a0235347fbd56c2"
server_key_iv: "ab97f12df4086bf8fc1faeb15ff0781a"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.345709 16921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "f9f2fe51b09140acbf1ed42d1c6452cc"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "4b06d754418e58eb8a0235347fbd56c2"
server_key_iv: "ab97f12df4086bf8fc1faeb15ff0781a"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.349233 16921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:14.351464 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.352629 16921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:14.352753 16921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "f9f2fe51b09140acbf1ed42d1c6452cc"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "4b06d754418e58eb8a0235347fbd56c2"
server_key_iv: "ab97f12df4086bf8fc1faeb15ff0781a"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.352854 16921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:14.387506 16921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:14.390202 16921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:14.390394 16921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:14.391819 16921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:14.391887 16921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.391950 16921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:14.391997 16921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:14.402524 16921 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:42691
I20260927 16:28:14.402657 17051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:42691 every 8 connection(s)
I20260927 16:28:14.403442 16921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:14.403908 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 16921
I20260927 16:28:14.404016 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:14.404340 15734 external_mini_cluster.cc:1428] Setting key 612cfd7e6ba472c1a0281f1e55977ce8
I20260927 16:28:14.405719 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.392540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:46763 (local address 127.15.93.190:42253)
0927 16:28:14.392701 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.392708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.393997 (+ 1289us) server_negotiation.cc:407] Connection header received
0927 16:28:14.395247 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.395252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.395315 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.395473 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.396676 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.397579 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.398934 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.399183 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.399297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.401925 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.401928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.401931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.401957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.403645 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.404201 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.404205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.404206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.404300 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.404637 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.404640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.404642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.404877 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.404976 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.405464 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.405559 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:28:14.406517 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.393385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.129:46763)
0927 16:28:14.393829 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.393877 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.394983 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.395685 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.395692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.396098 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.396546 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.396559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.397719 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.397725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.398811 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.398820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.399344 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.399348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.399535 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.400109 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.400125 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.401765 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.403762 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.403767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.403781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.404091 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.404401 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.404404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.404406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.404541 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.404970 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.404976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.405239 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.405426 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.405742 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260927 16:28:14.407735 17052 heartbeater.cc:347] Connected to a master server at 127.15.93.190:42253
I20260927 16:28:14.407872 17052 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:14.408308 17052 heartbeater.cc:511] Master 127.15.93.190:42253 requested a full tablet report, sending...
I20260927 16:28:14.409888 16848 ts_manager.cc:194] Registered new tserver with Master: f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691)
I20260927 16:28:14.411049 16848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:46763
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:14.421205 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.411717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47549 (local address 127.15.93.190:42253)
0927 16:28:14.411824 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.411828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.414571 (+ 2743us) server_negotiation.cc:407] Connection header received
0927 16:28:14.414641 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.414643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.414685 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.414759 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.415114 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.415733 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.416473 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.416861 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.416946 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.418237 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.418242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.418247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.418278 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.419697 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.420079 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.420085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.420087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.420167 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.420433 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.420438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.420439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.420598 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.420709 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.420948 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.421054 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 16:28:14.421214 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.411681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.129:47549)
0927 16:28:14.414485 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.414497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.414576 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.414783 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.414785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.414937 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.415025 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.415030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.415848 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.415850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.416390 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.416397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.416981 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.416986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.417110 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.417599 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.417618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.418076 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.419787 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.419790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.419791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.419975 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.420260 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.420264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.420266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.420344 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.420697 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.420700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.420787 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.420886 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.421017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:14.431522 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:42253
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:14.538066 17061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:14.538285 17061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:14.538324 17061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:14.542244 17061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:14.542297 17061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:14.542335 17061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:14.542395 17061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:14.547570 17061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:42253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:14.548584 17061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:14.549578 17061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:14.559180 17069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.559134 17067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.559144 17066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:14.559295 17061 server_base.cc:1034] running on GCE node
I20260927 16:28:14.559870 17061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:14.560496 17061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:14.561681 17061 hybrid_clock.cc:648] HybridClock initialized: now 1790526494561644 us; error 51 us; skew 500 ppm
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:14.564694 17061 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:14.565518 17061 webserver.cc:460] Webserver started at http://127.15.93.130:33659/ using document root <none> and password file <none>
I20260927 16:28:14.566066 17061 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:14.566169 17061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:14.566397 17061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:14.568492 17061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "b6c11152fa314148bf560fc1ebfc5864"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3bddfbcd10b80b86ffb6d041b7392856"
server_key_iv: "fe105ce9fae5dca6d6674c94c08c864e"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.569036 17061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "b6c11152fa314148bf560fc1ebfc5864"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3bddfbcd10b80b86ffb6d041b7392856"
server_key_iv: "fe105ce9fae5dca6d6674c94c08c864e"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.572602 17061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:14.574946 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.576229 17061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:14.576373 17061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "b6c11152fa314148bf560fc1ebfc5864"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "3bddfbcd10b80b86ffb6d041b7392856"
server_key_iv: "fe105ce9fae5dca6d6674c94c08c864e"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.576505 17061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:14.598201 17061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:14.600842 17061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:14.601046 17061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:14.602345 17061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:14.602415 17061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.602483 17061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:14.602540 17061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:14.612565 17061 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:36599
I20260927 16:28:14.612903 17191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:36599 every 8 connection(s)
I20260927 16:28:14.613644 17061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:14.615988 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.603152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55795 (local address 127.15.93.190:42253)
0927 16:28:14.603297 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.603302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.604166 (+ 864us) server_negotiation.cc:407] Connection header received
0927 16:28:14.605257 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.605261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.605323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.605434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.606881 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.607811 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.609107 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.609527 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.609640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.612553 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.612557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.612560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.612588 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.614192 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.614720 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.614722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.614723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.614800 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.615042 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.615044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.615046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.615202 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.615275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.615783 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.615875 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:28:14.616737 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.603570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.130:55795)
0927 16:28:14.604012 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.604057 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.605015 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.605619 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.605629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.606028 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.606707 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.606728 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.607950 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.607954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.608996 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.609006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.609666 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.609669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.609862 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.610462 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.610484 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.612388 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.614326 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.614333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.614348 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.614627 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.614864 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.614866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.614867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.614971 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.615288 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.615292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.615586 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.615735 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.615948 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260927 16:28:14.617311 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17061
I20260927 16:28:14.617426 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:14.617688 15734 external_mini_cluster.cc:1428] Setting key 11f7d1e73a9221acd59cfa6b9d13027c
I20260927 16:28:14.617915 17192 heartbeater.cc:347] Connected to a master server at 127.15.93.190:42253
I20260927 16:28:14.618054 17192 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:14.618553 17192 heartbeater.cc:511] Master 127.15.93.190:42253 requested a full tablet report, sending...
I20260927 16:28:14.619596 16849 ts_manager.cc:194] Registered new tserver with Master: b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599)
I20260927 16:28:14.620400 16849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:55795
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:14.628427 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.621019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.130:50699)
0927 16:28:14.621271 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.621287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.621370 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.621634 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.621644 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.621826 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.621912 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.621917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.622968 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.622971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.623574 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.623582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.624174 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.624176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.624240 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.624710 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.624721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.625011 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.626865 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.626867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.626869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.627040 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.627339 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.627341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.627342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.627476 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.627947 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.627949 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.628022 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.628190 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.628292 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
I20260927 16:28:14.628520 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.621339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:50699 (local address 127.15.93.190:42253)
0927 16:28:14.621449 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.621452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.621463 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:14.621493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.621495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.621535 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.621618 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.622023 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.622842 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.623689 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.624054 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.624187 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.625160 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.625166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.625172 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.625204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.626759 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.627145 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.627151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.627155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.627246 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.627571 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.627578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.627581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.627850 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.627952 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.628262 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.628381 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:14.643748 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:42253
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:14.752195 17201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:14.752440 17201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:14.752483 17201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:14.756479 17201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:14.756531 17201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:14.756572 17201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:14.756604 17201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:14.761756 17201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:42253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:14.762686 17201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:14.763628 17201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:14.773066 17209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.773067 17207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:14.773044 17206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:14.773290 17201 server_base.cc:1034] running on GCE node
I20260927 16:28:14.773713 17201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:14.774322 17201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:14.775487 17201 hybrid_clock.cc:648] HybridClock initialized: now 1790526494775474 us; error 24 us; skew 500 ppm
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:14.778072 17201 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:14.778877 17201 webserver.cc:460] Webserver started at http://127.15.93.131:37643/ using document root <none> and password file <none>
I20260927 16:28:14.779426 17201 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:14.779488 17201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:14.779662 17201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:14.781572 17201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "f387b41af15b46f1a636a51c3cacdc10"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "008181d930afed39da45908598a16c81"
server_key_iv: "bf8fce05be336f0dfcd320c5ad095533"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.782070 17201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "f387b41af15b46f1a636a51c3cacdc10"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "008181d930afed39da45908598a16c81"
server_key_iv: "bf8fce05be336f0dfcd320c5ad095533"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.785482 17201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:28:14.787689 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.788807 17201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:28:14.788894 17201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "f387b41af15b46f1a636a51c3cacdc10"
format_stamp: "Formatted at 2026-09-27 16:28:14 on dist-test-slave-grzf"
server_key: "008181d930afed39da45908598a16c81"
server_key_iv: "bf8fce05be336f0dfcd320c5ad095533"
server_key_version: "encryptionkey@0"
I20260927 16:28:14.788972 17201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:14.799134 17201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:14.801810 17201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:14.801954 17201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:14.803292 17201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:14.803342 17201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:14.803433 17201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:14.803467 17201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:14.813946 17201 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:41877
I20260927 16:28:14.814244 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:41877 every 8 connection(s)
I20260927 16:28:14.814972 17201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:14.817221 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.803996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:43797 (local address 127.15.93.190:42253)
0927 16:28:14.804157 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.804162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.805411 (+ 1249us) server_negotiation.cc:407] Connection header received
0927 16:28:14.806755 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.806758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.806802 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.806866 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.808527 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.809253 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.810783 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.811023 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.811164 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.813620 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.813624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.813627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.813654 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.815173 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.815760 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.815765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.815767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.815853 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.816190 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.816194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.816195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.816371 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.816473 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.816933 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.817048 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:28:14.817937 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.804774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.131:43797)
0927 16:28:14.805233 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.805296 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.806507 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.807073 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.807085 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.807639 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.808376 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.808398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.809370 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.809376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.810668 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.810678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.811190 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.811195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.811421 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.812034 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.812053 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.813462 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.815284 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.815290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.815303 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.815671 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.815950 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.815953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.815955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.816087 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.816470 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.816477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.816708 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.816879 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.817178 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260927 16:28:14.819006 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17201
I20260927 16:28:14.819136 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:14.819119 17332 heartbeater.cc:347] Connected to a master server at 127.15.93.190:42253
I20260927 16:28:14.819249 17332 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:14.819482 15734 external_mini_cluster.cc:1428] Setting key 2aababf31a85c713f06fbaafb28b46ab
I20260927 16:28:14.819815 17332 heartbeater.cc:511] Master 127.15.93.190:42253 requested a full tablet report, sending...
I20260927 16:28:14.820890 16848 ts_manager.cc:194] Registered new tserver with Master: f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877)
I20260927 16:28:14.821525 16848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:43797
I20260927 16:28:14.827944 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.821187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.15.93.131:52573)
0927 16:28:14.821448 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.821463 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.821543 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.822141 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.822143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.822301 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:14.822431 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.822437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.823422 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.823426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.823997 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.824006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.824440 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.824442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.824507 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.825003 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:14.825020 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:14.825319 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:14.826709 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.826712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.826714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.826907 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.827130 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:14.827132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:14.827133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:14.827203 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:14.827563 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:14.827565 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:14.827634 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.827713 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.827813 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260927 16:28:14.827944 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.821831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:52573 (local address 127.15.93.190:42253)
0927 16:28:14.821954 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.821957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.821967 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:28:14.821994 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.821996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.822027 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.822115 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.822540 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.823249 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.824109 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.824345 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.824427 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.825428 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.825430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.825432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.825448 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.826593 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.826987 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.826989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.826990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.827055 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.827280 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.827282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.827283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.827497 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.827570 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.827758 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.827838 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:28:14.833418 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:28:14 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:14.858146 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.850104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.15.93.190:42253)
0927 16:28:14.850227 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.850231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.850242 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:14.850280 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.850282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.850317 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.850407 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:14.851220 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.851818 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.852652 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.852865 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.852971 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.855418 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:14.855422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:14.855424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:14.855445 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:14.856691 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.857049 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.857051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.857053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.857120 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:14.857463 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:14.857465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:14.857466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:14.857624 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:14.857723 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.857929 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.858015 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:14.860831 16848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:14.863893 16848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:14.877684 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.871635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:36599 (local address 127.0.0.1:34676)
0927 16:28:14.872213 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.872262 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.872378 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.873382 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.873388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.873410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:14.873604 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.873612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.875029 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.875033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.876246 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.876257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.877268 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.877274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.877371 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.877388 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.877429 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.877512 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260927 16:28:14.877785 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.872167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.15.93.131:41877)
0927 16:28:14.873061 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.873072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.873093 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:28:14.873162 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.873168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.873332 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.873786 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:14.873900 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.874614 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.876182 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.876998 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.877107 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.877292 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.877542 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.877636 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":101,"threads_started":1}
I20260927 16:28:14.877857 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.872727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.15.93.129:42691)
0927 16:28:14.873099 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.873114 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.873136 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:28:14.873212 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.873217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.873416 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.873603 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:14.873783 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.874912 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.876411 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.876942 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.877068 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.877232 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.877590 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.877683 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":84,"threads_started":1}
I20260927 16:28:14.877691 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.871769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41877 (local address 127.0.0.1:38322)
0927 16:28:14.872289 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.872305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.872394 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.873474 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.873477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.873492 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:14.873635 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.873641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.874723 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.874728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.876059 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.876068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.877308 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.877313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.877404 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.877423 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.877493 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.877568 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":66,"thread_start_us":90,"threads_started":1}
I20260927 16:28:14.878326 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.872315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:42691 (local address 127.0.0.1:55486)
0927 16:28:14.872727 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.872740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.872821 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.873591 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.873594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.873606 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:14.873695 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.873703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.875155 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.875158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.876301 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.876312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.877068 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.877071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.877182 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.877225 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.877329 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.877447 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":88,"threads_started":1}
I20260927 16:28:14.878578 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.871763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.15.93.130:36599)
0927 16:28:14.872811 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.872823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.872847 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:28:14.872927 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.872933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.873158 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.873390 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:14.873739 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.874921 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.876372 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.877137 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.878139 (+ 1002us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.878304 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.878357 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.878447 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":107,"threads_started":1}
I20260927 16:28:14.881124 17126 tablet_service.cc:1467] Processing CreateTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa (DEFAULT_TABLE table=test-table [id=31c04e3875f740959e168992389d579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:14.881135 17266 tablet_service.cc:1467] Processing CreateTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa (DEFAULT_TABLE table=test-table [id=31c04e3875f740959e168992389d579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:14.881127 16986 tablet_service.cc:1467] Processing CreateTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa (DEFAULT_TABLE table=test-table [id=31c04e3875f740959e168992389d579f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:14.882063 17126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899aa556330d4f359c84b1ca8e3e47aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:14.882084 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899aa556330d4f359c84b1ca8e3e47aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:14.882095 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899aa556330d4f359c84b1ca8e3e47aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:14.888496 17355 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Bootstrap starting.
I20260927 16:28:14.889896 17356 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Bootstrap starting.
I20260927 16:28:14.890041 17357 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Bootstrap starting.
I20260927 16:28:14.890753 17355 tablet_bootstrap.cc:654] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:14.891630 17355 log.cc:824] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:14.891727 17357 tablet_bootstrap.cc:654] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:14.891949 17356 tablet_bootstrap.cc:654] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:14.892494 17357 log.cc:824] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:14.892762 17356 log.cc:824] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:14.893810 17355 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: No bootstrap required, opened a new log
I20260927 16:28:14.893989 17355 ts_tablet_manager.cc:1397] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:28:14.894513 17357 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: No bootstrap required, opened a new log
I20260927 16:28:14.894629 17356 tablet_bootstrap.cc:492] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: No bootstrap required, opened a new log
I20260927 16:28:14.894702 17357 ts_tablet_manager.cc:1397] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:28:14.894796 17356 ts_tablet_manager.cc:1397] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:14.897439 17357 raft_consensus.cc:348] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.897676 17357 raft_consensus.cc:374] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:14.897737 17357 raft_consensus.cc:729] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f387b41af15b46f1a636a51c3cacdc10, State: Initialized, Role: FOLLOWER
I20260927 16:28:14.897847 17355 raft_consensus.cc:348] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.898096 17355 raft_consensus.cc:374] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:14.898154 17355 raft_consensus.cc:729] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c11152fa314148bf560fc1ebfc5864, State: Initialized, Role: FOLLOWER
I20260927 16:28:14.898176 17357 consensus_queue.cc:260] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.898574 17355 consensus_queue.cc:260] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.898922 17332 heartbeater.cc:503] Master 127.15.93.190:42253 was elected leader, sending a full tablet report...
I20260927 16:28:14.899243 17357 ts_tablet_manager.cc:1428] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 16:28:14.899479 17192 heartbeater.cc:503] Master 127.15.93.190:42253 was elected leader, sending a full tablet report...
I20260927 16:28:14.898577 17356 raft_consensus.cc:348] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.899528 17355 ts_tablet_manager.cc:1428] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 16:28:14.899555 17356 raft_consensus.cc:374] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:14.899609 17356 raft_consensus.cc:729] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f2fe51b09140acbf1ed42d1c6452cc, State: Initialized, Role: FOLLOWER
I20260927 16:28:14.899988 17356 consensus_queue.cc:260] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.900615 17052 heartbeater.cc:503] Master 127.15.93.190:42253 was elected leader, sending a full tablet report...
I20260927 16:28:14.900887 17356 ts_tablet_manager.cc:1428] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 16:28:14.905401 17053 tablet.cc:2369] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:14.970817 17362 raft_consensus.cc:482] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:14.970997 17362 raft_consensus.cc:504] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.972033 17362 leader_election.cc:288] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877), f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691)
I20260927 16:28:14.975574 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.972374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:34461 (local address 127.15.93.131:41877)
0927 16:28:14.972497 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.972502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.972516 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:14.972595 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.972598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.972648 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.972734 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:14.972992 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.973504 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.974595 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.974998 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.975126 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.975191 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.975395 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.975463 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:28:14.975574 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.972270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41877 (local address 127.15.93.130:34461)
0927 16:28:14.972394 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.972411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.972505 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.972746 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.972749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.972767 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:14.972879 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.972884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.973612 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.973619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.974463 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.974483 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.975118 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.975122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.975245 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.975262 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.975345 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.975445 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 16:28:14.976161 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.972368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:36587 (local address 127.15.93.129:42691)
0927 16:28:14.972502 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:14.972510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:14.972770 (+ 260us) server_negotiation.cc:407] Connection header received
0927 16:28:14.972925 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:14.972927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:14.972966 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:14.973039 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:14.973657 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.974131 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.975111 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.975570 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.975707 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.975751 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:14.975988 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 16:28:14.976055 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:28:14.976181 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:14.972270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:42691 (local address 127.15.93.130:36587)
0927 16:28:14.972682 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:14.972693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:14.972821 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:14.973059 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:14.973061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:14.973076 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:14.973535 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.973542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.974241 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.974244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.974999 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:14.975007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:14.975705 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:14.975709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:14.975813 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:14.975836 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:28:14.975967 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 16:28:14.976036 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":111,"thread_start_us":118,"threads_started":1}
I20260927 16:28:14.976317 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899aa556330d4f359c84b1ca8e3e47aa" candidate_uuid: "b6c11152fa314148bf560fc1ebfc5864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f387b41af15b46f1a636a51c3cacdc10" is_pre_election: true
I20260927 16:28:14.976613 17286 raft_consensus.cc:2454] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c11152fa314148bf560fc1ebfc5864 in term 0.
I20260927 16:28:14.976753 17006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899aa556330d4f359c84b1ca8e3e47aa" candidate_uuid: "b6c11152fa314148bf560fc1ebfc5864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" is_pre_election: true
I20260927 16:28:14.977016 17006 raft_consensus.cc:2454] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c11152fa314148bf560fc1ebfc5864 in term 0.
I20260927 16:28:14.977093 17077 leader_election.cc:302] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c11152fa314148bf560fc1ebfc5864, f387b41af15b46f1a636a51c3cacdc10; no voters:
I20260927 16:28:14.977491 17362 raft_consensus.cc:2781] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:14.977622 17362 raft_consensus.cc:482] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:14.977684 17362 raft_consensus.cc:3037] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:14.978790 17362 raft_consensus.cc:504] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.979143 17362 leader_election.cc:288] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [CANDIDATE]: Term 1 election: Requested vote from peers f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877), f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691)
I20260927 16:28:14.979512 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899aa556330d4f359c84b1ca8e3e47aa" candidate_uuid: "b6c11152fa314148bf560fc1ebfc5864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f387b41af15b46f1a636a51c3cacdc10"
I20260927 16:28:14.979545 17006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899aa556330d4f359c84b1ca8e3e47aa" candidate_uuid: "b6c11152fa314148bf560fc1ebfc5864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc"
I20260927 16:28:14.979658 17006 raft_consensus.cc:3037] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:14.979655 17286 raft_consensus.cc:3037] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:14.981122 17006 raft_consensus.cc:2454] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c11152fa314148bf560fc1ebfc5864 in term 1.
I20260927 16:28:14.981379 17286 raft_consensus.cc:2454] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c11152fa314148bf560fc1ebfc5864 in term 1.
I20260927 16:28:14.981451 17080 leader_election.cc:302] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c11152fa314148bf560fc1ebfc5864, f9f2fe51b09140acbf1ed42d1c6452cc; no voters:
I20260927 16:28:14.981628 17362 raft_consensus.cc:2781] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:14.981860 17362 raft_consensus.cc:686] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 1 LEADER]: Becoming Leader. State: Replica: b6c11152fa314148bf560fc1ebfc5864, State: Running, Role: LEADER
I20260927 16:28:14.982179 17362 consensus_queue.cc:237] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } }
I20260927 16:28:14.985390 16848 catalog_manager.cc:5495] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "b6c11152fa314148bf560fc1ebfc5864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:15.006081 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.002395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.15.93.130:36599)
0927 16:28:15.002547 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.002551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.002566 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:15.002783 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.002786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.002846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.002951 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:15.003238 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.003761 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.004627 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.004864 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.004984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.005309 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.005684 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.005806 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.005885 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.005941 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:28:15.011476 17286 raft_consensus.cc:1261] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Refusing update from remote peer b6c11152fa314148bf560fc1ebfc5864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.011476 17006 raft_consensus.cc:1261] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Refusing update from remote peer b6c11152fa314148bf560fc1ebfc5864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.012252 17365 consensus_queue.cc:1035] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.012481 17362 consensus_queue.cc:1035] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.019290 17370 mvcc.cc:205] Tried to move back new op lower bound from 7333996523560132608 to 7333996523449556992. Current Snapshot: MvccSnapshot[applied={T|T < 7333996523560132608}]
I20260927 16:28:15.021004 17369 mvcc.cc:205] Tried to move back new op lower bound from 7333996523560132608 to 7333996523449556992. Current Snapshot: MvccSnapshot[applied={T|T < 7333996523560132608}]
I20260927 16:28:15.029342 16849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:15.029500 16849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:15.031688 16849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Sending DeleteTablet for 3 replicas of tablet 899aa556330d4f359c84b1ca8e3e47aa
I20260927 16:28:15.032389 17126 tablet_service.cc:1514] Processing DeleteTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:34676
I20260927 16:28:15.032380 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38322
I20260927 16:28:15.032428 16986 tablet_service.cc:1514] Processing DeleteTablet for tablet 899aa556330d4f359c84b1ca8e3e47aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:55486
I20260927 16:28:15.032850 17378 tablet_replica.cc:330] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: stopping tablet replica
I20260927 16:28:15.033000 17380 tablet_replica.cc:330] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: stopping tablet replica
I20260927 16:28:15.033128 17378 raft_consensus.cc:2229] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:15.033373 17380 raft_consensus.cc:2229] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:15.033422 17378 raft_consensus.cc:2258] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:15.033622 16849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:28:15.033741 17380 raft_consensus.cc:2258] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:28:15.033998 16849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:15.034945 17378 ts_tablet_manager.cc:1905] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.035343 17380 ts_tablet_manager.cc:1905] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.037676 17378 ts_tablet_manager.cc:1918] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:15.037773 17378 log.cc:1196] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/899aa556330d4f359c84b1ca8e3e47aa
I20260927 16:28:15.038074 17380 ts_tablet_manager.cc:1918] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:15.038126 17378 ts_tablet_manager.cc:1939] T 899aa556330d4f359c84b1ca8e3e47aa P f387b41af15b46f1a636a51c3cacdc10: Deleting consensus metadata
I20260927 16:28:15.038259 17380 log.cc:1196] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/899aa556330d4f359c84b1ca8e3e47aa
I20260927 16:28:15.038697 17380 ts_tablet_manager.cc:1939] T 899aa556330d4f359c84b1ca8e3e47aa P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting consensus metadata
I20260927 16:28:15.039327 17379 tablet_replica.cc:330] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: stopping tablet replica
I20260927 16:28:15.039618 17379 raft_consensus.cc:2229] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:15.039604 16986 tablet_service.cc:1467] Processing CreateTablet for tablet c57a1c2986fd476699491072450314e4 (DEFAULT_TABLE table=test-table [id=dbe253fca8664d7aaefa82be48c73501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:15.039834 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57a1c2986fd476699491072450314e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.039983 17379 raft_consensus.cc:2258] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:15.040081 17126 tablet_service.cc:1467] Processing CreateTablet for tablet c57a1c2986fd476699491072450314e4 (DEFAULT_TABLE table=test-table [id=dbe253fca8664d7aaefa82be48c73501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:15.040313 17126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57a1c2986fd476699491072450314e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.040576 16836 catalog_manager.cc:4841] TS f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691): tablet 899aa556330d4f359c84b1ca8e3e47aa (table test-table [id=31c04e3875f740959e168992389d579f]) successfully deleted
I20260927 16:28:15.041569 17379 ts_tablet_manager.cc:1905] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.042081 17356 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Bootstrap starting.
I20260927 16:28:15.042709 17355 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Bootstrap starting.
I20260927 16:28:15.042927 16833 catalog_manager.cc:4841] TS f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877): tablet 899aa556330d4f359c84b1ca8e3e47aa (table test-table [id=31c04e3875f740959e168992389d579f]) successfully deleted
I20260927 16:28:15.043040 17356 tablet_bootstrap.cc:654] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.043701 17355 tablet_bootstrap.cc:654] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.044185 17379 ts_tablet_manager.cc:1918] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:15.044284 17379 log.cc:1196] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/899aa556330d4f359c84b1ca8e3e47aa
I20260927 16:28:15.044454 17266 tablet_service.cc:1467] Processing CreateTablet for tablet c57a1c2986fd476699491072450314e4 (DEFAULT_TABLE table=test-table [id=dbe253fca8664d7aaefa82be48c73501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:15.044605 17379 ts_tablet_manager.cc:1939] T 899aa556330d4f359c84b1ca8e3e47aa P b6c11152fa314148bf560fc1ebfc5864: Deleting consensus metadata
I20260927 16:28:15.044698 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57a1c2986fd476699491072450314e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.045452 16836 catalog_manager.cc:4841] TS b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599): tablet 899aa556330d4f359c84b1ca8e3e47aa (table test-table [id=31c04e3875f740959e168992389d579f]) successfully deleted
I20260927 16:28:15.046360 17356 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: No bootstrap required, opened a new log
I20260927 16:28:15.046442 17356 ts_tablet_manager.cc:1397] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:28:15.046574 17355 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: No bootstrap required, opened a new log
I20260927 16:28:15.046646 17355 ts_tablet_manager.cc:1397] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:28:15.046772 17357 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Bootstrap starting.
I20260927 16:28:15.046862 17356 raft_consensus.cc:348] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.046976 17356 raft_consensus.cc:374] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.047010 17356 raft_consensus.cc:729] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f2fe51b09140acbf1ed42d1c6452cc, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.047065 17355 raft_consensus.cc:348] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.047163 17355 raft_consensus.cc:374] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.047204 17355 raft_consensus.cc:729] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c11152fa314148bf560fc1ebfc5864, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.047312 17355 consensus_queue.cc:260] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.047389 17356 consensus_queue.cc:260] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.047601 17355 ts_tablet_manager.cc:1428] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:15.047648 17357 tablet_bootstrap.cc:654] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.048800 17356 ts_tablet_manager.cc:1428] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:15.049525 17357 tablet_bootstrap.cc:492] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: No bootstrap required, opened a new log
I20260927 16:28:15.049597 17357 ts_tablet_manager.cc:1397] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:15.049997 17357 raft_consensus.cc:348] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.050087 17357 raft_consensus.cc:374] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.050117 17357 raft_consensus.cc:729] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f387b41af15b46f1a636a51c3cacdc10, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.050241 17357 consensus_queue.cc:260] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.050453 17357 ts_tablet_manager.cc:1428] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 16:28:15.065914 17333 tablet.cc:2369] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:28:15.114711 17193 tablet.cc:2369] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:15.172894 17361 raft_consensus.cc:482] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:15.173086 17361 raft_consensus.cc:504] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.174451 17361 leader_election.cc:288] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691), b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599)
I20260927 16:28:15.178913 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.174909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:36599 (local address 127.15.93.131:54433)
0927 16:28:15.175291 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.175302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.175445 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.175749 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.175751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.175768 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:15.175889 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.175895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.176745 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.176748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.177616 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.177624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.178474 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.178478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.178577 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.178603 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.178681 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.178747 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260927 16:28:15.178920 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.175105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:54433 (local address 127.15.93.130:36599)
0927 16:28:15.175220 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.175224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.175396 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:28:15.175571 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.175577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.175629 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.175725 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:15.175996 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.176624 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.177740 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.178345 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.178467 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.178533 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.178710 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.178786 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:15.179173 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.174764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38195 (local address 127.15.93.129:42691)
0927 16:28:15.174908 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.174912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.174929 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:28:15.175077 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.175080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.175131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.175219 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:15.175614 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.176363 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.177856 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.178436 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.178531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.178587 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.178970 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.179048 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:15.179512 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.174661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:42691 (local address 127.15.93.131:38195)
0927 16:28:15.174814 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.174831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.174933 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.175334 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.175337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.175356 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:15.175500 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.175506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.176467 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.176470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.177739 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.177749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.178608 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.178612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.178751 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.178773 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.178844 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.178910 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":45}
I20260927 16:28:15.179769 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57a1c2986fd476699491072450314e4" candidate_uuid: "f387b41af15b46f1a636a51c3cacdc10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c11152fa314148bf560fc1ebfc5864" is_pre_election: true
I20260927 16:28:15.179952 17006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57a1c2986fd476699491072450314e4" candidate_uuid: "f387b41af15b46f1a636a51c3cacdc10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" is_pre_election: true
I20260927 16:28:15.179987 17146 raft_consensus.cc:2454] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f387b41af15b46f1a636a51c3cacdc10 in term 0.
I20260927 16:28:15.180066 17006 raft_consensus.cc:2454] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f387b41af15b46f1a636a51c3cacdc10 in term 0.
I20260927 16:28:15.180388 17220 leader_election.cc:302] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c11152fa314148bf560fc1ebfc5864, f387b41af15b46f1a636a51c3cacdc10; no voters:
I20260927 16:28:15.180658 17361 raft_consensus.cc:2781] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:15.180732 17361 raft_consensus.cc:482] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:15.180781 17361 raft_consensus.cc:3037] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.181777 17361 raft_consensus.cc:504] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.182101 17361 leader_election.cc:288] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [CANDIDATE]: Term 1 election: Requested vote from peers f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691), b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599)
I20260927 16:28:15.182523 17006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57a1c2986fd476699491072450314e4" candidate_uuid: "f387b41af15b46f1a636a51c3cacdc10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc"
I20260927 16:28:15.182624 17006 raft_consensus.cc:3037] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.182749 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57a1c2986fd476699491072450314e4" candidate_uuid: "f387b41af15b46f1a636a51c3cacdc10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c11152fa314148bf560fc1ebfc5864"
I20260927 16:28:15.182878 17146 raft_consensus.cc:3037] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.183601 17006 raft_consensus.cc:2454] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f387b41af15b46f1a636a51c3cacdc10 in term 1.
I20260927 16:28:15.183861 17220 leader_election.cc:302] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f387b41af15b46f1a636a51c3cacdc10, f9f2fe51b09140acbf1ed42d1c6452cc; no voters:
I20260927 16:28:15.184003 17146 raft_consensus.cc:2454] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f387b41af15b46f1a636a51c3cacdc10 in term 1.
I20260927 16:28:15.184042 17361 raft_consensus.cc:2781] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:15.184259 17361 raft_consensus.cc:686] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 1 LEADER]: Becoming Leader. State: Replica: f387b41af15b46f1a636a51c3cacdc10, State: Running, Role: LEADER
I20260927 16:28:15.184501 17361 consensus_queue.cc:237] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.186743 16849 catalog_manager.cc:5495] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "f387b41af15b46f1a636a51c3cacdc10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } health_report { overall_health: HEALTHY } } }
Sep 27 16:28:15 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:15.238361 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.224945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.15.93.190:42253)
0927 16:28:15.225092 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.225097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.225202 (+ 105us) server_negotiation.cc:407] Connection header received
0927 16:28:15.225302 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.225305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.225345 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.225411 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:15.226009 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.226498 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.227231 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.227490 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.227565 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.235485 (+ 7920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:15.235488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:15.235490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:15.235510 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:15.236925 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.237313 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.237315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.237316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.237382 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.237595 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.237597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.237598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.237762 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:15.237831 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.238101 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.238230 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":73}
I20260927 16:28:15.238363 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.224856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42253 (local address 127.0.0.1:55462)
0927 16:28:15.225112 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.225135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.225229 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.225446 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.225448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.225826 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:15.225916 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.225922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.226587 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.226589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.227142 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.227149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.227581 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.227583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.227645 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.233889 (+ 6244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:15.233901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:15.235336 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:15.237009 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:15.237016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:15.237021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:15.237235 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:15.237455 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:15.237457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:15.237458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:15.237526 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:15.237844 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:15.237849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:15.237941 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.238025 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.238179 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260927 16:28:15.241921 16849 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:55462:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:28:15.242605 16849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:15.247002 17126 tablet_service.cc:1467] Processing CreateTablet for tablet c272fe7936244e3ba248329d4632df16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0871430eeaca454088ea17d307eca49a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:15.246981 16986 tablet_service.cc:1467] Processing CreateTablet for tablet c272fe7936244e3ba248329d4632df16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0871430eeaca454088ea17d307eca49a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:15.247071 17266 tablet_service.cc:1467] Processing CreateTablet for tablet c272fe7936244e3ba248329d4632df16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0871430eeaca454088ea17d307eca49a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:15.247376 17126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c272fe7936244e3ba248329d4632df16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.247400 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c272fe7936244e3ba248329d4632df16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.247437 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c272fe7936244e3ba248329d4632df16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:15.249959 17355 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: Bootstrap starting.
I20260927 16:28:15.250185 17356 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: Bootstrap starting.
I20260927 16:28:15.250509 17357 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: Bootstrap starting.
I20260927 16:28:15.250839 17355 tablet_bootstrap.cc:654] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.251123 17356 tablet_bootstrap.cc:654] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.251503 17357 tablet_bootstrap.cc:654] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:15.252063 17355 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: No bootstrap required, opened a new log
I20260927 16:28:15.252157 17355 ts_tablet_manager.cc:1397] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:15.252404 17356 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: No bootstrap required, opened a new log
I20260927 16:28:15.252496 17356 ts_tablet_manager.cc:1397] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 16:28:15.252578 17355 raft_consensus.cc:348] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.252702 17355 raft_consensus.cc:374] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.252746 17355 raft_consensus.cc:729] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c11152fa314148bf560fc1ebfc5864, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.252905 17355 consensus_queue.cc:260] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.253000 17356 raft_consensus.cc:348] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.253113 17356 raft_consensus.cc:374] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.253153 17356 raft_consensus.cc:729] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f2fe51b09140acbf1ed42d1c6452cc, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.253219 17355 ts_tablet_manager.cc:1428] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:15.253192 17375 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.253288 17357 tablet_bootstrap.cc:492] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: No bootstrap required, opened a new log
I20260927 16:28:15.253266 17356 consensus_queue.cc:260] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.253331 17375 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.253357 17357 ts_tablet_manager.cc:1397] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:28:15.253504 17363 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.253634 17363 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.253513 17356 ts_tablet_manager.cc:1428] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:15.253808 17357 raft_consensus.cc:348] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.253917 17357 raft_consensus.cc:374] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:15.253947 17357 raft_consensus.cc:729] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f387b41af15b46f1a636a51c3cacdc10, State: Initialized, Role: FOLLOWER
I20260927 16:28:15.254184 17357 consensus_queue.cc:260] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.254503 17357 ts_tablet_manager.cc:1428] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:15.254994 17361 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.255128 17361 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.310077 17363 raft_consensus.cc:482] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:15.310261 17363 raft_consensus.cc:504] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.311578 17363 leader_election.cc:288] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599), f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877)
I20260927 16:28:15.315224 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.311850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:44591 (local address 127.15.93.130:36599)
0927 16:28:15.311970 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.311976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.312235 (+ 259us) server_negotiation.cc:407] Connection header received
0927 16:28:15.312355 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.312357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.312397 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.312460 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:15.312736 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.313243 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314279 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.314690 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314816 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.314872 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.315038 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.315110 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:28:15.315245 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.311738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:36599 (local address 127.15.93.129:44591)
0927 16:28:15.312134 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.312150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.312260 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.312495 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.312497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.312511 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:15.312614 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.312618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.313350 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.313354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.314156 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.314165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.314806 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.314914 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.314937 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.315023 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.315090 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":109,"threads_started":1}
I20260927 16:28:15.315416 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.311810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41877 (local address 127.15.93.129:38103)
0927 16:28:15.312250 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.312262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.312354 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.312740 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.312743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.312760 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:15.312876 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.312881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.313566 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.313570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.314324 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.314332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.314964 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.315067 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.315089 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.315184 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.315251 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":88,"threads_started":1}
I20260927 16:28:15.315402 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.311891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:38103 (local address 127.15.93.131:41877)
0927 16:28:15.312009 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.312015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.312538 (+ 523us) server_negotiation.cc:407] Connection header received
0927 16:28:15.312585 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.312587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.312622 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.312724 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:15.312979 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.313456 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314454 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.314845 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.314968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.315014 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.315201 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.315267 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:15.315657 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c272fe7936244e3ba248329d4632df16" candidate_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c11152fa314148bf560fc1ebfc5864" is_pre_election: true
I20260927 16:28:15.315786 17146 raft_consensus.cc:2454] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2fe51b09140acbf1ed42d1c6452cc in term 0.
I20260927 16:28:15.315915 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c272fe7936244e3ba248329d4632df16" candidate_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f387b41af15b46f1a636a51c3cacdc10" is_pre_election: true
I20260927 16:28:15.316071 17286 raft_consensus.cc:2454] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2fe51b09140acbf1ed42d1c6452cc in term 0.
I20260927 16:28:15.316141 16940 leader_election.cc:302] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c11152fa314148bf560fc1ebfc5864, f9f2fe51b09140acbf1ed42d1c6452cc; no voters:
I20260927 16:28:15.316413 17363 raft_consensus.cc:2781] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:15.316470 17363 raft_consensus.cc:482] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:15.316509 17363 raft_consensus.cc:3037] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.317467 17363 raft_consensus.cc:504] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.317831 17363 leader_election.cc:288] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [CANDIDATE]: Term 1 election: Requested vote from peers b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599), f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877)
I20260927 16:28:15.318123 17146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c272fe7936244e3ba248329d4632df16" candidate_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c11152fa314148bf560fc1ebfc5864"
I20260927 16:28:15.318157 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c272fe7936244e3ba248329d4632df16" candidate_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f387b41af15b46f1a636a51c3cacdc10"
I20260927 16:28:15.318235 17286 raft_consensus.cc:3037] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.318257 17146 raft_consensus.cc:3037] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:15.319285 17286 raft_consensus.cc:2454] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2fe51b09140acbf1ed42d1c6452cc in term 1.
I20260927 16:28:15.319355 17146 raft_consensus.cc:2454] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2fe51b09140acbf1ed42d1c6452cc in term 1.
I20260927 16:28:15.319603 16937 leader_election.cc:302] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f387b41af15b46f1a636a51c3cacdc10, f9f2fe51b09140acbf1ed42d1c6452cc; no voters:
I20260927 16:28:15.319785 17363 raft_consensus.cc:2781] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:15.319962 17363 raft_consensus.cc:686] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 LEADER]: Becoming Leader. State: Replica: f9f2fe51b09140acbf1ed42d1c6452cc, State: Running, Role: LEADER
I20260927 16:28:15.320240 17363 consensus_queue.cc:237] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } }
I20260927 16:28:15.321568 17398 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: TxnStatusTablet state changed. Reason: New leader f9f2fe51b09140acbf1ed42d1c6452cc. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.321683 17398 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:15.321954 17400 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:28:15.322104 17400 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:28:15.322666 16848 catalog_manager.cc:5495] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc reported cstate change: term changed from 0 to 1, leader changed from <none> to f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129). New cstate: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:15.335048 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.330847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:42691 (local address 127.0.0.1:55498)
0927 16:28:15.331069 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.331083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.331202 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.331541 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.331544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.331551 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:15.331661 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.331669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.332366 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.332370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.333350 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.333362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.333934 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.333937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.334033 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.334088 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.334743 (+ 655us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.334760 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.334827 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.334916 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 16:28:15.335048 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.330928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.15.93.129:42691)
0927 16:28:15.331042 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.331046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.331150 (+ 104us) server_negotiation.cc:407] Connection header received
0927 16:28:15.331324 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.331326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.331421 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.331509 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:15.331761 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.332262 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.333498 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.333809 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.333935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.334245 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.334633 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.334769 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.334847 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.334900 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:28:15.338016 17286 raft_consensus.cc:1261] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Refusing update from remote peer f9f2fe51b09140acbf1ed42d1c6452cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.338232 17146 raft_consensus.cc:1261] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Refusing update from remote peer f9f2fe51b09140acbf1ed42d1c6452cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.338466 17398 consensus_queue.cc:1035] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.338788 17363 consensus_queue.cc:1035] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.339792 17361 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: TxnStatusTablet state changed. Reason: New leader f9f2fe51b09140acbf1ed42d1c6452cc. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.339903 17361 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.341627 17375 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: TxnStatusTablet state changed. Reason: New leader f9f2fe51b09140acbf1ed42d1c6452cc. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.341740 17375 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.342864 17363 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.342972 17363 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:15.343530 17368 mvcc.cc:205] Tried to move back new op lower bound from 7333996524899536896 to 7333996524834615296. Current Snapshot: MvccSnapshot[applied={T|T < 7333996524899536896 or (T in {7333996524899536896})}]
I20260927 16:28:15.343788 17375 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.343878 17375 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P b6c11152fa314148bf560fc1ebfc5864: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.344240 17363 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.344357 17363 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f9f2fe51b09140acbf1ed42d1c6452cc: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:15.344732 17361 tablet_replica.cc:439] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 } } peers { permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 } } peers { permanent_uuid: "f387b41af15b46f1a636a51c3cacdc10" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41877 } } }
I20260927 16:28:15.344831 17361 tablet_replica.cc:442] T c272fe7936244e3ba248329d4632df16 P f387b41af15b46f1a636a51c3cacdc10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:15.352088 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.348384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.15.93.131:41877)
0927 16:28:15.348498 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.348501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.348532 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:28:15.348686 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.348689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.348735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.348813 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:15.349051 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.349617 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.350726 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.350924 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.350994 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.351276 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.351706 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.351788 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.351892 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.351934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:28:15.362002 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.358492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:33953 (local address 127.15.93.129:42691)
0927 16:28:15.358608 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.358612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.358832 (+ 220us) server_negotiation.cc:407] Connection header received
0927 16:28:15.358970 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.358973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.359010 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.359066 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:15.359309 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.359839 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.361039 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.361248 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.361329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.361632 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.361691 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.361751 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.361869 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.361907 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:28:15.362076 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.358412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:42691 (local address 127.15.93.131:33953)
0927 16:28:15.358757 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.358771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.358871 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.359095 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.359098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.359105 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:15.359213 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.359219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.359939 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.359942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.360920 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.360930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.361353 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.361356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.361448 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.361496 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.361778 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.361785 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.361854 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.361927 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260927 16:28:15.369060 17006 raft_consensus.cc:1261] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Refusing update from remote peer f387b41af15b46f1a636a51c3cacdc10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.369086 17146 raft_consensus.cc:1261] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Refusing update from remote peer f387b41af15b46f1a636a51c3cacdc10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:15.369547 17361 consensus_queue.cc:1035] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6c11152fa314148bf560fc1ebfc5864" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.369938 17385 consensus_queue.cc:1035] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f2fe51b09140acbf1ed42d1c6452cc" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:15.390795 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.386967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41877 (local address 127.15.93.129:57249)
0927 16:28:15.387231 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:15.387244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:15.387328 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:15.387586 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:15.387588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:15.387594 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:15.387704 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.387708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.388691 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.388695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.389597 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:15.389611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.390174 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.390177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:15.390240 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.390282 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.390550 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.390555 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:28:15.390609 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 16:28:15.390667 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260927 16:28:15.390791 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.387001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:57249 (local address 127.15.93.131:41877)
0927 16:28:15.387115 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.387119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.387312 (+ 193us) server_negotiation.cc:407] Connection header received
0927 16:28:15.387450 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.387453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.387500 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.387580 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:15.387794 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.388587 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.389711 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.390046 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.390146 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.390396 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:15.390472 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:15.390543 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.390618 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.390667 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 16:28:15.419173 16847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:15.419345 16847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:15.421888 16847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d535f9aff44c008662088863a6a39e: Sending DeleteTablet for 3 replicas of tablet c57a1c2986fd476699491072450314e4
I20260927 16:28:15.422554 17126 tablet_service.cc:1514] Processing DeleteTablet for tablet c57a1c2986fd476699491072450314e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:34676
I20260927 16:28:15.422807 17379 tablet_replica.cc:330] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: stopping tablet replica
I20260927 16:28:15.422999 17379 raft_consensus.cc:2229] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:15.423219 17379 raft_consensus.cc:2258] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:15.423370 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet c57a1c2986fd476699491072450314e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38322
I20260927 16:28:15.424259 17378 tablet_replica.cc:330] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: stopping tablet replica
I20260927 16:28:15.424489 17378 raft_consensus.cc:2229] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:15.424753 17378 raft_consensus.cc:2258] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:15.424930 16986 tablet_service.cc:1514] Processing DeleteTablet for tablet c57a1c2986fd476699491072450314e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:15 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:55486
I20260927 16:28:15.425120 17380 tablet_replica.cc:330] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: stopping tablet replica
I20260927 16:28:15.425213 17380 raft_consensus.cc:2229] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:15.425326 17380 raft_consensus.cc:2258] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:15.426023 17379 ts_tablet_manager.cc:1905] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.426105 17380 ts_tablet_manager.cc:1905] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.428717 17380 ts_tablet_manager.cc:1918] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:15.428717 17379 ts_tablet_manager.cc:1918] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:15.428784 17380 log.cc:1196] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/c57a1c2986fd476699491072450314e4
I20260927 16:28:15.428784 17379 log.cc:1196] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/c57a1c2986fd476699491072450314e4
I20260927 16:28:15.429041 17380 ts_tablet_manager.cc:1939] T c57a1c2986fd476699491072450314e4 P f9f2fe51b09140acbf1ed42d1c6452cc: Deleting consensus metadata
I20260927 16:28:15.429059 17379 ts_tablet_manager.cc:1939] T c57a1c2986fd476699491072450314e4 P b6c11152fa314148bf560fc1ebfc5864: Deleting consensus metadata
I20260927 16:28:15.429723 16836 catalog_manager.cc:4841] TS f9f2fe51b09140acbf1ed42d1c6452cc (127.15.93.129:42691): tablet c57a1c2986fd476699491072450314e4 (table test-table [id=dbe253fca8664d7aaefa82be48c73501]) successfully deleted
I20260927 16:28:15.429876 16836 catalog_manager.cc:4841] TS b6c11152fa314148bf560fc1ebfc5864 (127.15.93.130:36599): tablet c57a1c2986fd476699491072450314e4 (table test-table [id=dbe253fca8664d7aaefa82be48c73501]) successfully deleted
I20260927 16:28:15.430508 17378 ts_tablet_manager.cc:1905] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:15.436801 17378 ts_tablet_manager.cc:1918] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:15.436885 17378 log.cc:1196] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/c57a1c2986fd476699491072450314e4
I20260927 16:28:15.437134 17378 ts_tablet_manager.cc:1939] T c57a1c2986fd476699491072450314e4 P f387b41af15b46f1a636a51c3cacdc10: Deleting consensus metadata
I20260927 16:28:15.437806 16833 catalog_manager.cc:4841] TS f387b41af15b46f1a636a51c3cacdc10 (127.15.93.131:41877): tablet c57a1c2986fd476699491072450314e4 (table test-table [id=dbe253fca8664d7aaefa82be48c73501]) successfully deleted
Sep 27 16:28:15 dist-test-slave-grzf krb5kdc[16804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526494, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:15.451975 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.439206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.15.93.129:42691)
0927 16:28:15.439308 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.439313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.439325 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:15.439363 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.439385 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.439429 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.439508 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:15.440419 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.441197 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.442392 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.442724 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.442788 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.447106 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:15.447116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:15.447133 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:15.447162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:15.449390 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.450145 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.450153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.450157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.450261 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.450617 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.450621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.450622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.450959 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:15.451146 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.451658 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.451800 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 16:28:15.452855 17046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55506
I20260927 16:28:15.464424 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:15.454528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.15.93.190:42253)
0927 16:28:15.454656 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:15.454661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:15.454789 (+ 128us) server_negotiation.cc:407] Connection header received
0927 16:28:15.454911 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:15.454914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:15.454961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:15.455054 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:15.455921 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.456860 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.458106 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:15.458516 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:15.458626 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:15.460429 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:15.460433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:15.460436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:15.460461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:15.462202 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.462754 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.462758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.462760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.462851 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:15.463204 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:15.463209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:15.463213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:15.463520 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:15.463650 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:15.464059 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 16:28:15.464201 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 16:28:15.465269 16847 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55464
I20260927 16:28:15.466777 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16921
I20260927 16:28:15.476698 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17061
I20260927 16:28:15.483424 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17201
I20260927 16:28:15.493067 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 16817
2026-09-27T16:28:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2722 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:15 dist-test-slave-grzf krb5kdc[17423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:15 dist-test-slave-grzf krb5kdc[17423](info): set up 2 sockets
Sep 27 16:28:15 dist-test-slave-grzf krb5kdc[17423](info): commencing operation
krb5kdc: starting...
W20260927 16:28:16.523448 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:16 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:16.636346 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:39705
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:16.742482 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:16.742708 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:16.742748 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:16.746771 17436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:16.746824 17436 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:28:16.746840 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:16.746893 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:16.746923 17436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:16.746953 17436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:16.752153 17436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:39705
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:16.753094 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:16.754050 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:16.761961 17441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:16.762001 17442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:16.761961 17444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:16.762161 17436 server_base.cc:1034] running on GCE node
I20260927 16:28:16.762912 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:16.764034 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:16.765218 17436 hybrid_clock.cc:648] HybridClock initialized: now 1790526496765185 us; error 46 us; skew 500 ppm
Sep 27 16:28:16 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526496, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:16.767968 17436 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:16.768784 17436 webserver.cc:460] Webserver started at http://127.15.93.190:38927/ using document root <none> and password file <none>
I20260927 16:28:16.769321 17436 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:16.769411 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:16.769639 17436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:16.771665 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "e740df81e0344d9c9306b32b6f2a7a5b"
format_stamp: "Formatted at 2026-09-27 16:28:16 on dist-test-slave-grzf"
server_key: "478d8335cb36984f50cce727c310b37d"
server_key_iv: "3e11e239aae4b1520653dff59c9f559b"
server_key_version: "encryptionkey@0"
I20260927 16:28:16.772217 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "e740df81e0344d9c9306b32b6f2a7a5b"
format_stamp: "Formatted at 2026-09-27 16:28:16 on dist-test-slave-grzf"
server_key: "478d8335cb36984f50cce727c310b37d"
server_key_iv: "3e11e239aae4b1520653dff59c9f559b"
server_key_version: "encryptionkey@0"
I20260927 16:28:16.775763 17436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:16.777946 17451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:16.779084 17436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:16.779223 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "e740df81e0344d9c9306b32b6f2a7a5b"
format_stamp: "Formatted at 2026-09-27 16:28:16 on dist-test-slave-grzf"
server_key: "478d8335cb36984f50cce727c310b37d"
server_key_iv: "3e11e239aae4b1520653dff59c9f559b"
server_key_version: "encryptionkey@0"
I20260927 16:28:16.779330 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:16.810694 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:16.814049 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:16.814258 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:16.822132 17436 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:39705
I20260927 16:28:16.822208 17513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:39705 every 8 connection(s)
I20260927 16:28:16.823073 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:16.825800 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:16.831773 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Bootstrap starting.
I20260927 16:28:16.831894 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17436
I20260927 16:28:16.832059 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:16.832542 15734 external_mini_cluster.cc:1428] Setting key 6da7a91fe11cb2657ae6cd0de93a9957
I20260927 16:28:16.834661 17514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:16.835605 17514 log.cc:824] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:16.837597 17514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: No bootstrap required, opened a new log
I20260927 16:28:16.841696 17514 raft_consensus.cc:348] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } }
I20260927 16:28:16.841928 17514 raft_consensus.cc:374] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:16.841995 17514 raft_consensus.cc:729] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e740df81e0344d9c9306b32b6f2a7a5b, State: Initialized, Role: FOLLOWER
I20260927 16:28:16.842450 17514 consensus_queue.cc:260] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } }
I20260927 16:28:16.842566 17514 raft_consensus.cc:388] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:16.842644 17514 raft_consensus.cc:482] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:16.842734 17514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:16.843763 17514 raft_consensus.cc:504] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } }
Sep 27 16:28:16 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:16.844138 17514 leader_election.cc:302] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e740df81e0344d9c9306b32b6f2a7a5b; no voters:
I20260927 16:28:16.844481 17514 leader_election.cc:288] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:16.844595 17519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:16.844805 17519 raft_consensus.cc:686] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [term 1 LEADER]: Becoming Leader. State: Replica: e740df81e0344d9c9306b32b6f2a7a5b, State: Running, Role: LEADER
I20260927 16:28:16.845125 17519 consensus_queue.cc:237] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } }
I20260927 16:28:16.845750 17514 sys_catalog.cc:564] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:16.846776 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } } }
I20260927 16:28:16.846889 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:16.847285 17521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e740df81e0344d9c9306b32b6f2a7a5b. Latest consensus state: current_term: 1 leader_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e740df81e0344d9c9306b32b6f2a7a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 39705 } } }
I20260927 16:28:16.847563 17521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:16.850481 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:16.834037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.15.93.190:39705)
0927 16:28:16.836227 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:16.836248 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:16.836291 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:28:16.837025 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:16.837044 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:16.837358 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:16.837697 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:16.838537 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:16.840269 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:16.841127 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:16.841623 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:16.841799 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:16.844727 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:16.844733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:16.844750 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:16.844781 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:16.847325 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:16.847929 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:16.847936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:16.847942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:16.848062 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:16.848543 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:16.848546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:16.848548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:16.848914 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:16.849115 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:16.849413 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 16:28:16.849723 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2035,"thread_start_us":136,"threads_started":1}
W20260927 16:28:16.850807 17534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:16.850874 17534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:16.851130 17528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:16.851998 17528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:16.858256 17528 catalog_manager.cc:1347] Generated new cluster ID: fa3ac926764f492ea4f6fd4d05658098
I20260927 16:28:16.858321 17528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:16.894550 17528 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:16.895589 17528 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:16.923862 17528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Generated new TSK 0
I20260927 16:28:16.924639 17528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:16.952277 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:39705
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:17.059562 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:17.059794 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:17.059839 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:17.063726 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:17.063782 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:17.063854 17540 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:28:17.063889 17540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:17.063941 17540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:17.069205 17540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:39705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:17.070232 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:17.071215 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:17.080613 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:17.080616 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:17.080613 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:17.080893 17540 server_base.cc:1034] running on GCE node
I20260927 16:28:17.081394 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:17.082026 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:17.083181 17540 hybrid_clock.cc:648] HybridClock initialized: now 1790526497083149 us; error 44 us; skew 500 ppm
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:17.085727 17540 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:17.086539 17540 webserver.cc:460] Webserver started at http://127.15.93.129:37497/ using document root <none> and password file <none>
I20260927 16:28:17.087096 17540 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:17.087164 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:17.087325 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:17.089323 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "8ae7d005542b00e77b449dda83c034ed"
server_key_iv: "16db86d4e3872b6b544a4bf1f454ee57"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.089824 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "8ae7d005542b00e77b449dda83c034ed"
server_key_iv: "16db86d4e3872b6b544a4bf1f454ee57"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.093384 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:17.095666 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.096796 17540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:28:17.096923 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "8ae7d005542b00e77b449dda83c034ed"
server_key_iv: "16db86d4e3872b6b544a4bf1f454ee57"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.097025 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:17.123087 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:17.126075 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:17.126284 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:17.127810 17540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:17.127876 17540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.127950 17540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:17.127992 17540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:17.138983 17540 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:32935
I20260927 16:28:17.139053 17670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:32935 every 8 connection(s)
I20260927 16:28:17.139920 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:17.141726 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.128623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:60367 (local address 127.15.93.190:39705)
0927 16:28:17.128749 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.128752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.129777 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 16:28:17.130758 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.130765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.130831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.130895 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.132574 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.133485 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.134916 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.135155 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.135273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.137766 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.137771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.137775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.137807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.139538 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.140042 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.140048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.140051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.140170 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.140603 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.140606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.140608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.140832 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.140943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.141419 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.141548 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:17.142429 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.129158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.129:60367)
0927 16:28:17.129611 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.129665 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.130500 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.131356 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.131365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.131826 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.132357 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.132371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.133647 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.133650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.134787 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.134797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.135327 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.135331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.135532 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.136100 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.136119 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.137555 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.139656 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.139662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.139676 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.139927 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.140286 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.140290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.140292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.140482 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.140939 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.140945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.141186 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.141365 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.141667 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260927 16:28:17.143726 17671 heartbeater.cc:347] Connected to a master server at 127.15.93.190:39705
I20260927 16:28:17.143862 17671 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:17.144399 17671 heartbeater.cc:511] Master 127.15.93.190:39705 requested a full tablet report, sending...
I20260927 16:28:17.146132 17467 ts_manager.cc:194] Registered new tserver with Master: b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935)
I20260927 16:28:17.147585 17467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:60367
I20260927 16:28:17.148137 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17540
I20260927 16:28:17.148257 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:17.148583 15734 external_mini_cluster.cc:1428] Setting key a0cdfa2f7e012acd516eb7f0a9ea1ec7
I20260927 16:28:17.154848 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.145802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.129:34919)
0927 16:28:17.146252 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.146267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.146362 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.146701 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.146704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.146922 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.147036 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.147044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.148957 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.148959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.149647 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.149656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.150103 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.150107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.150223 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.150718 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.150733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.151228 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.153334 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.153337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.153339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.153532 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.153893 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.153895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.153896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.153969 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.154421 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.154423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.154507 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.154585 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.154697 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":49,"thread_start_us":95,"threads_started":1}
I20260927 16:28:17.155185 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.145912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:34919 (local address 127.15.93.190:39705)
0927 16:28:17.146078 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.146082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.146345 (+ 263us) server_negotiation.cc:407] Connection header received
0927 16:28:17.146458 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.146461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.146602 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.146687 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.147149 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.148211 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.149762 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.149992 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.150099 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.151430 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.151435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.151444 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.151466 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.153211 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.153682 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.153688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.153693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.153791 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.154060 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.154066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.154071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.154325 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.154435 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.154696 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.154841 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:17.174546 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:39705
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:17.279619 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:17.279842 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:17.279934 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:17.283933 17680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:17.283980 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:17.284052 17680 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:28:17.284085 17680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:17.284145 17680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:17.289211 17680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:39705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:17.290135 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:17.291066 17680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:17.299959 17686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:17.300101 17680 server_base.cc:1034] running on GCE node
W20260927 16:28:17.299979 17688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:17.300024 17685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:17.300776 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:17.301374 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:17.302560 17680 hybrid_clock.cc:648] HybridClock initialized: now 1790526497302497 us; error 74 us; skew 500 ppm
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:17.305276 17680 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:17.306082 17680 webserver.cc:460] Webserver started at http://127.15.93.130:42629/ using document root <none> and password file <none>
I20260927 16:28:17.306576 17680 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:17.306630 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:17.306787 17680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:17.308735 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "0205afccb32440f28ef9d9348ee6f2d3"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "d400df2626cfde4274de741c036df5d5"
server_key_iv: "2f83d96ca6058a1eaba66af66dfc12b6"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.309237 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "0205afccb32440f28ef9d9348ee6f2d3"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "d400df2626cfde4274de741c036df5d5"
server_key_iv: "2f83d96ca6058a1eaba66af66dfc12b6"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.312687 17680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:17.314845 17695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.316052 17680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:17.316176 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "0205afccb32440f28ef9d9348ee6f2d3"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "d400df2626cfde4274de741c036df5d5"
server_key_iv: "2f83d96ca6058a1eaba66af66dfc12b6"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.316303 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:17.333570 17680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:17.337028 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:17.337190 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:17.338594 17680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:17.338683 17680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.338778 17680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:17.338830 17680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:17.349129 17680 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:44633
I20260927 16:28:17.349156 17810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:44633 every 8 connection(s)
I20260927 16:28:17.350128 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:17.353073 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.339320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55845 (local address 127.15.93.190:39705)
0927 16:28:17.339492 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.339497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.340888 (+ 1391us) server_negotiation.cc:407] Connection header received
0927 16:28:17.342263 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.342266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.342313 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.342420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.343774 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.344714 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.346374 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.346622 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.346738 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.349684 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.349688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.349691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.349724 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.351331 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.351847 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.351849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.351850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.351927 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.352207 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.352209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.352210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.352372 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.352443 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.352862 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.352955 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 16:28:17.353782 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.340223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.130:55845)
0927 16:28:17.340706 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.340766 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.342036 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.342598 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.342605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.342943 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.343594 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.343617 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.344848 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.344852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.346238 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.346248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.346773 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.346777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.346993 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.347741 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.347764 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.349505 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.351493 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.351497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.351507 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.351753 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.352001 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.352003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.352004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.352122 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.352453 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.352457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.352666 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.352803 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.353024 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260927 16:28:17.354911 17811 heartbeater.cc:347] Connected to a master server at 127.15.93.190:39705
I20260927 16:28:17.355063 17811 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:17.355594 17811 heartbeater.cc:511] Master 127.15.93.190:39705 requested a full tablet report, sending...
I20260927 16:28:17.356578 17468 ts_manager.cc:194] Registered new tserver with Master: 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633)
I20260927 16:28:17.357360 17468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:55845
I20260927 16:28:17.360314 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17680
I20260927 16:28:17.360417 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:17.360702 15734 external_mini_cluster.cc:1428] Setting key fe2af50c0ce5f4685ef45e362947dfff
I20260927 16:28:17.364303 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.356969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:46519 (local address 127.15.93.190:39705)
0927 16:28:17.357651 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.357654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.357664 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:28:17.357689 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.357691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.357719 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.357807 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.358224 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.358711 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.359575 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.359795 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.359902 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.360986 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.360989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.360991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.361010 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.362571 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.362980 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.362986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.362990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.363076 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.363321 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.363327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.363332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.363668 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.363780 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.364114 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.364201 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624}
I20260927 16:28:17.364460 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.356853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.130:46519)
0927 16:28:17.357123 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.357137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.357224 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.357823 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.357826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.358013 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.358104 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.358110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.358829 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.358835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.359464 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.359473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.359906 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.359909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.360003 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.360500 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.360513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.360857 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.362704 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.362707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.362708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.362888 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.363166 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.363168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.363169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.363237 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.363814 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.363818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.363929 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.364055 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.364235 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:17.385673 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:39705
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:17.491773 17820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:17.491992 17820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:17.492036 17820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:17.495971 17820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:17.496021 17820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:17.496058 17820 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 16:28:17.496104 17820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:17.496168 17820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:17.501323 17820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:39705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:17.502269 17820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:17.503214 17820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:17.512310 17826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:17.512293 17828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:17.512298 17825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:17.512609 17820 server_base.cc:1034] running on GCE node
I20260927 16:28:17.512974 17820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:17.513593 17820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:17.514775 17820 hybrid_clock.cc:648] HybridClock initialized: now 1790526497514741 us; error 46 us; skew 500 ppm
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:17.517549 17820 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:17.518316 17820 webserver.cc:460] Webserver started at http://127.15.93.131:46517/ using document root <none> and password file <none>
I20260927 16:28:17.518849 17820 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:17.518934 17820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:17.519158 17820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:17.521270 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "ecee55870cc4f1f5a500f0447c9afab3"
server_key_iv: "3aca50d5172a12afc4977ef8b9f9d8c4"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.521821 17820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "ecee55870cc4f1f5a500f0447c9afab3"
server_key_iv: "3aca50d5172a12afc4977ef8b9f9d8c4"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.525230 17820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:17.527595 17835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.528702 17820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:17.528831 17820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
format_stamp: "Formatted at 2026-09-27 16:28:17 on dist-test-slave-grzf"
server_key: "ecee55870cc4f1f5a500f0447c9afab3"
server_key_iv: "3aca50d5172a12afc4977ef8b9f9d8c4"
server_key_version: "encryptionkey@0"
I20260927 16:28:17.528936 17820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:17.545334 17820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:17.548676 17820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:17.548887 17820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:17.550266 17820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:17.550334 17820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:17.550398 17820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:17.550449 17820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:17.562114 17820 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:38459
I20260927 16:28:17.562160 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:38459 every 8 connection(s)
I20260927 16:28:17.563208 17820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:17.564842 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.551102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:49199 (local address 127.15.93.190:39705)
0927 16:28:17.551230 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.551234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.552352 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 16:28:17.553507 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.553513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.553583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.553688 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.555048 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.555742 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.557069 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.557294 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.557411 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.561022 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.561027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.561033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.561064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.562681 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.563152 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.563158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.563163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.563262 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.563719 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.563722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.563724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.563969 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.564063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.564620 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.564725 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:17.565586 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.551711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.131:49199)
0927 16:28:17.552205 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.552247 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.553224 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.553915 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.553925 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.554374 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.554903 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.554917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.555866 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.555872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.556944 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.556953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.557453 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.557457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.557689 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.558511 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.558567 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.560850 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.562791 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.562795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.562806 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.563042 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.563400 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.563404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.563406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.563595 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.564115 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.564120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.564382 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.564539 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.564771 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20260927 16:28:17.566978 17951 heartbeater.cc:347] Connected to a master server at 127.15.93.190:39705
I20260927 16:28:17.567147 17951 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:17.567658 17951 heartbeater.cc:511] Master 127.15.93.190:39705 requested a full tablet report, sending...
I20260927 16:28:17.568665 17468 ts_manager.cc:194] Registered new tserver with Master: 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:17.569433 17468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:49199
I20260927 16:28:17.571070 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 17820
I20260927 16:28:17.571206 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:17.571563 15734 external_mini_cluster.cc:1428] Setting key c6c47fad26eedbdf8f2ada6e56b0d099
I20260927 16:28:17.573593 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:28:17.576468 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.569127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38393 (local address 127.15.93.190:39705)
0927 16:28:17.569236 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.569240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.569344 (+ 104us) server_negotiation.cc:407] Connection header received
0927 16:28:17.569446 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.569448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.569481 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.569574 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.570008 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.570506 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.571592 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.571896 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.572011 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.573038 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.573041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.573044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.573067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.574770 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.575174 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.575180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.575185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.575271 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.575600 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.575606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.575610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.575855 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.575945 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.576222 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.576320 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:17.576485 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.568996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.15.93.131:38393)
0927 16:28:17.569261 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.569275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.569369 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.569605 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.569608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.569812 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.569903 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.569909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.570631 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.570635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.571469 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.571491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.572012 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.572016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.572105 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.572576 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.572590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.572926 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.574892 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.574895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.574896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.575079 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.575362 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.575364 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.575365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.575510 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.575951 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.575956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.576047 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.576168 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.576309 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:17.597995 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.590528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.15.93.190:39705)
0927 16:28:17.590648 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.590652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.590664 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:17.590702 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.590704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.590739 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.590809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.591499 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.592021 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.592833 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.593029 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.593099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.595507 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.595511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.595513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.595533 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.596688 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.597044 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.597046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.597047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.597114 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.597356 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.597358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.597359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.597509 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.597603 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.597793 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.597881 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:28:17.600653 17468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:17.603657 17468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:17.617096 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.611263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44633 (local address 127.0.0.1:53670)
0927 16:28:17.612383 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.612396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.612488 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.612944 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.612946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.612958 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.613209 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.613215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614242 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.614248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.615521 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.615531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.616581 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.616687 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.616733 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.616839 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.616951 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":83,"thread_start_us":71,"threads_started":1}
I20260927 16:28:17.617128 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.611863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.15.93.130:44633)
0927 16:28:17.612262 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.612274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.612494 (+ 220us) server_negotiation.cc:407] Connection header received
0927 16:28:17.612580 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.612585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.612785 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.612969 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.613353 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614122 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.615650 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.616449 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616555 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.616719 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.616883 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.616979 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":105,"threads_started":1}
I20260927 16:28:17.617403 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.611181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:38459 (local address 127.0.0.1:35976)
0927 16:28:17.611606 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.611653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.611769 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.612886 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.612891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.612908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.613203 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.613211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614679 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.614682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.615995 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.616003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.617012 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.617016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.617104 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.617133 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.617208 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.617272 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":160,"thread_start_us":64,"threads_started":1}
I20260927 16:28:17.617573 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.611900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32935 (local address 127.0.0.1:33518)
0927 16:28:17.612688 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.612700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.612773 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.613151 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.613154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.613171 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.613336 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.613343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614745 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.614747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.615905 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.615915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.617146 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.617152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.617270 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.617292 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.617357 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.617436 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":30,"thread_start_us":89,"threads_started":1}
I20260927 16:28:17.617852 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.611268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.15.93.131:38459)
0927 16:28:17.612262 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.612273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.612297 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:28:17.612375 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.612380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.612589 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.612741 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.613350 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614544 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616103 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.616693 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.616955 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.617573 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.617699 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":118,"threads_started":1}
I20260927 16:28:17.617973 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.612101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.15.93.129:32935)
0927 16:28:17.612356 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.612370 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.612779 (+ 409us) server_negotiation.cc:407] Connection header received
0927 16:28:17.612863 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.612870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.613037 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.613178 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.613463 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.614621 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616036 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.616835 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.616933 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.617070 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.617756 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.617847 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
I20260927 16:28:17.620361 17605 tablet_service.cc:1467] Processing CreateTablet for tablet b64838a15fa046939aff687083a9b927 (DEFAULT_TABLE table=test-table [id=5cbafe759c0c42708199e5c44e9389d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.620386 17885 tablet_service.cc:1467] Processing CreateTablet for tablet b64838a15fa046939aff687083a9b927 (DEFAULT_TABLE table=test-table [id=5cbafe759c0c42708199e5c44e9389d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.620386 17745 tablet_service.cc:1467] Processing CreateTablet for tablet b64838a15fa046939aff687083a9b927 (DEFAULT_TABLE table=test-table [id=5cbafe759c0c42708199e5c44e9389d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.621307 17605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64838a15fa046939aff687083a9b927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.621345 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64838a15fa046939aff687083a9b927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.621338 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64838a15fa046939aff687083a9b927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.627175 17974 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Bootstrap starting.
I20260927 16:28:17.628512 17975 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Bootstrap starting.
I20260927 16:28:17.629448 17974 tablet_bootstrap.cc:654] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.630033 17976 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Bootstrap starting.
I20260927 16:28:17.630292 17974 log.cc:824] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:17.630548 17975 tablet_bootstrap.cc:654] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.631577 17975 log.cc:824] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:17.631923 17976 tablet_bootstrap.cc:654] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.632436 17974 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: No bootstrap required, opened a new log
I20260927 16:28:17.632622 17974 ts_tablet_manager.cc:1397] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:28:17.632714 17976 log.cc:824] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:17.633543 17975 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: No bootstrap required, opened a new log
I20260927 16:28:17.633733 17975 ts_tablet_manager.cc:1397] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:17.634544 17976 tablet_bootstrap.cc:492] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: No bootstrap required, opened a new log
I20260927 16:28:17.634719 17976 ts_tablet_manager.cc:1397] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:28:17.636660 17974 raft_consensus.cc:348] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.636909 17974 raft_consensus.cc:374] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.636965 17974 raft_consensus.cc:729] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.637034 17975 raft_consensus.cc:348] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.637284 17975 raft_consensus.cc:374] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.637351 17975 raft_consensus.cc:729] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf8e7ea9474e8e88b6e60c592e772c, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.637406 17974 consensus_queue.cc:260] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.637779 17975 consensus_queue.cc:260] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.638111 17671 heartbeater.cc:503] Master 127.15.93.190:39705 was elected leader, sending a full tablet report...
I20260927 16:28:17.638368 17974 ts_tablet_manager.cc:1428] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:28:17.638398 17976 raft_consensus.cc:348] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.638612 17976 raft_consensus.cc:374] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.638650 17976 raft_consensus.cc:729] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.638677 17975 ts_tablet_manager.cc:1428] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:28:17.638984 17976 consensus_queue.cc:260] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.639088 17951 heartbeater.cc:503] Master 127.15.93.190:39705 was elected leader, sending a full tablet report...
I20260927 16:28:17.639595 17811 heartbeater.cc:503] Master 127.15.93.190:39705 was elected leader, sending a full tablet report...
I20260927 16:28:17.639863 17976 ts_tablet_manager.cc:1428] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 16:28:17.641109 17672 tablet.cc:2369] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:17.718809 17980 raft_consensus.cc:482] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:17.719048 17980 raft_consensus.cc:504] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.720228 17980 leader_election.cc:288] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459), 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633)
I20260927 16:28:17.723987 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.720446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44633 (local address 127.15.93.129:42105)
0927 16:28:17.720748 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.720763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.720870 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.721180 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.721182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.721198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.721333 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.721339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.722049 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.722052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.722873 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.722881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.723544 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.723549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.723658 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.723689 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.723761 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.723816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":6,"thread_start_us":124,"threads_started":1}
I20260927 16:28:17.724004 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.720637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:42105 (local address 127.15.93.130:44633)
0927 16:28:17.720767 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.720772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.720850 (+ 78us) server_negotiation.cc:407] Connection header received
0927 16:28:17.721008 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.721014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.721073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.721131 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.721440 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.721945 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.722999 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.723429 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.723521 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.723598 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.723789 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.723876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:28:17.724082 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.720577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:54761 (local address 127.15.93.131:38459)
0927 16:28:17.720749 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.720754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.720912 (+ 158us) server_negotiation.cc:407] Connection header received
0927 16:28:17.721057 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.721059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.721097 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.721186 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.721438 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.721952 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.723027 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.723487 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.723600 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.723662 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.723889 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.723952 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 16:28:17.724056 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.720436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:38459 (local address 127.15.93.129:54761)
0927 16:28:17.720816 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.720832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.720955 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.721217 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.721221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.721240 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.721336 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.721341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.722082 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.722085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.722883 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.722891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.723613 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.723620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.723761 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.723788 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.723861 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.723924 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":42,"thread_start_us":84,"threads_started":1}
I20260927 16:28:17.724785 17765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64838a15fa046939aff687083a9b927" candidate_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0205afccb32440f28ef9d9348ee6f2d3" is_pre_election: true
I20260927 16:28:17.724825 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64838a15fa046939aff687083a9b927" candidate_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" is_pre_election: true
I20260927 16:28:17.725083 17765 raft_consensus.cc:2454] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e64e9fd4a5438f85cc16430ac3c46f in term 0.
I20260927 16:28:17.725083 17905 raft_consensus.cc:2454] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e64e9fd4a5438f85cc16430ac3c46f in term 0.
I20260927 16:28:17.725544 17557 leader_election.cc:302] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf8e7ea9474e8e88b6e60c592e772c, b3e64e9fd4a5438f85cc16430ac3c46f; no voters:
I20260927 16:28:17.725829 17980 raft_consensus.cc:2781] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:17.725908 17980 raft_consensus.cc:482] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:17.725939 17980 raft_consensus.cc:3037] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.726963 17980 raft_consensus.cc:504] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.727325 17980 leader_election.cc:288] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [CANDIDATE]: Term 1 election: Requested vote from peers 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459), 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633)
I20260927 16:28:17.727645 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64838a15fa046939aff687083a9b927" candidate_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
I20260927 16:28:17.727694 17765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64838a15fa046939aff687083a9b927" candidate_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0205afccb32440f28ef9d9348ee6f2d3"
I20260927 16:28:17.727772 17905 raft_consensus.cc:3037] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.727788 17765 raft_consensus.cc:3037] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.729044 17765 raft_consensus.cc:2454] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e64e9fd4a5438f85cc16430ac3c46f in term 1.
I20260927 16:28:17.729044 17905 raft_consensus.cc:2454] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e64e9fd4a5438f85cc16430ac3c46f in term 1.
I20260927 16:28:17.729321 17556 leader_election.cc:302] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, b3e64e9fd4a5438f85cc16430ac3c46f; no voters:
I20260927 16:28:17.729476 17980 raft_consensus.cc:2781] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:17.729648 17980 raft_consensus.cc:686] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 LEADER]: Becoming Leader. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Running, Role: LEADER
I20260927 16:28:17.729976 17980 consensus_queue.cc:237] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } }
I20260927 16:28:17.733175 17468 catalog_manager.cc:5495] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129). New cstate: current_term: 1 leader_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } health_report { overall_health: HEALTHY } } }
Sep 27 16:28:17 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526496, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:17.769063 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.761100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:39705 (local address 127.0.0.1:35568)
0927 16:28:17.761378 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.761391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.761496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.761722 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.761724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.762100 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:17.762208 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.762214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.762921 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.762923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.763496 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.763503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.763885 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.763888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.763945 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.764533 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:17.764544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:17.765859 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:17.767662 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.767670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.767674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.767889 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.768153 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:17.768155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:17.768156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:17.768229 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:17.768578 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:17.768583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:17.768682 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.768771 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.768887 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
I20260927 16:28:17.769068 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.761131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.15.93.190:39705)
0927 16:28:17.761272 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.761277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.761464 (+ 187us) server_negotiation.cc:407] Connection header received
0927 16:28:17.761573 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.761576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.761616 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.761686 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:17.762303 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.762834 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.763590 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.763792 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.763870 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.766032 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:17.766036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:17.766038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:17.766058 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:17.767544 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.767991 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.767995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.767996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.768070 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:17.768306 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:17.768308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:17.768309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:17.768468 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:17.768567 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.768825 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.768940 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":49}
I20260927 16:28:17.773037 17468 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35568:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:28:17.773763 17468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:17.778512 17605 tablet_service.cc:1467] Processing CreateTablet for tablet df791a67cf18410a87589c5dc57e7e66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da338beaa1034978b5cb72722aeaf28e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:17.778600 17745 tablet_service.cc:1467] Processing CreateTablet for tablet df791a67cf18410a87589c5dc57e7e66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da338beaa1034978b5cb72722aeaf28e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:17.778878 17885 tablet_service.cc:1467] Processing CreateTablet for tablet df791a67cf18410a87589c5dc57e7e66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da338beaa1034978b5cb72722aeaf28e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:17.779286 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df791a67cf18410a87589c5dc57e7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.779286 17605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df791a67cf18410a87589c5dc57e7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.779426 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df791a67cf18410a87589c5dc57e7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.781688 17974 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: Bootstrap starting.
I20260927 16:28:17.782133 17975 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: Bootstrap starting.
I20260927 16:28:17.783123 17975 tablet_bootstrap.cc:654] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.783123 17974 tablet_bootstrap.cc:654] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.783818 17976 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: Bootstrap starting.
I20260927 16:28:17.784518 17975 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: No bootstrap required, opened a new log
I20260927 16:28:17.784601 17975 ts_tablet_manager.cc:1397] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:17.784809 17976 tablet_bootstrap.cc:654] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.784909 17974 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: No bootstrap required, opened a new log
I20260927 16:28:17.785000 17974 ts_tablet_manager.cc:1397] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:17.785018 17975 raft_consensus.cc:348] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.785101 17975 raft_consensus.cc:374] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.785121 17975 raft_consensus.cc:729] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf8e7ea9474e8e88b6e60c592e772c, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.785260 17975 consensus_queue.cc:260] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.785455 17974 raft_consensus.cc:348] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.785562 17974 raft_consensus.cc:374] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.785601 17974 raft_consensus.cc:729] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.785562 17981 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.785707 17981 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.785722 17974 consensus_queue.cc:260] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.786005 17974 ts_tablet_manager.cc:1428] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:17.785964 17980 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.786080 17980 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.786681 17976 tablet_bootstrap.cc:492] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: No bootstrap required, opened a new log
I20260927 16:28:17.786778 17976 ts_tablet_manager.cc:1397] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:17.787285 17976 raft_consensus.cc:348] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.787441 17976 raft_consensus.cc:374] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.787490 17976 raft_consensus.cc:729] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.787624 17976 consensus_queue.cc:260] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.787904 17976 ts_tablet_manager.cc:1428] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:17.785576 17975 ts_tablet_manager.cc:1428] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:17.787941 17982 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.788075 17982 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 16:28:17.814184 17952 tablet.cc:2369] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:17.841161 17982 raft_consensus.cc:482] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:17.841289 17982 raft_consensus.cc:504] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.842481 17982 leader_election.cc:288] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:17.846505 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.842667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32935 (local address 127.15.93.130:55149)
0927 16:28:17.842886 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.842902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.842987 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.843438 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.843444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.843472 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.843604 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.843612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.844541 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.844548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.845376 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.845385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.846055 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.846059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.846190 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.846215 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.846290 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.846356 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 16:28:17.846550 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.842788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55149 (local address 127.15.93.129:32935)
0927 16:28:17.842925 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.842929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.842947 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:28:17.843169 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.843174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.843223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.843294 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.843746 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.844440 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.845487 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.845946 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.846036 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.846075 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.846318 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.846413 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:28:17.846706 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.842975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37877 (local address 127.15.93.131:38459)
0927 16:28:17.843158 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.843164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.843179 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:17.843316 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.843320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.843401 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.843494 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:17.843917 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.844515 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.845660 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.846060 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.846201 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.846247 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.846507 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.846587 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:28:17.846737 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.842715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:38459 (local address 127.15.93.130:37877)
0927 16:28:17.843090 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.843103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.843208 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.843609 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.843612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.843636 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:17.843775 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.843780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.844614 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.844617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.845544 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.845554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.846254 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.846257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.846390 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.846412 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.846477 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.846589 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":21,"thread_start_us":93,"threads_started":1}
I20260927 16:28:17.847090 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df791a67cf18410a87589c5dc57e7e66" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" is_pre_election: true
I20260927 16:28:17.847242 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df791a67cf18410a87589c5dc57e7e66" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" is_pre_election: true
I20260927 16:28:17.847339 17625 raft_consensus.cc:2454] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:17.847395 17905 raft_consensus.cc:2454] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:17.847756 17697 leader_election.cc:302] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c; no voters:
I20260927 16:28:17.848047 17982 raft_consensus.cc:2781] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:17.848104 17982 raft_consensus.cc:482] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:17.848157 17982 raft_consensus.cc:3037] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.849108 17982 raft_consensus.cc:504] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.849475 17982 leader_election.cc:288] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Requested vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:17.849752 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df791a67cf18410a87589c5dc57e7e66" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
I20260927 16:28:17.849808 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df791a67cf18410a87589c5dc57e7e66" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
I20260927 16:28:17.849888 17625 raft_consensus.cc:3037] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.849926 17905 raft_consensus.cc:3037] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:17.850848 17905 raft_consensus.cc:2454] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:17.850905 17625 raft_consensus.cc:2454] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:17.851192 17697 leader_election.cc:302] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c; no voters:
I20260927 16:28:17.851434 17982 raft_consensus.cc:2781] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 16:28:17.851512 17812 tablet.cc:2369] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:17.851665 17982 raft_consensus.cc:686] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Running, Role: LEADER
I20260927 16:28:17.851984 17982 consensus_queue.cc:237] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.854077 17997 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: TxnStatusTablet state changed. Reason: New leader 0205afccb32440f28ef9d9348ee6f2d3. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.854197 17997 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:17.854450 17999 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:28:17.854678 17999 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:28:17.855027 17468 catalog_manager.cc:5495] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:17.868433 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.864750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.15.93.130:44633)
0927 16:28:17.864884 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.864893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.864986 (+ 93us) server_negotiation.cc:407] Connection header received
0927 16:28:17.865192 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.865199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.865251 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.865349 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:17.865582 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.866074 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.867000 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.867228 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.867336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.867699 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.868055 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.868182 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.868254 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.868301 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:28:17.868436 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.864602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44633 (local address 127.0.0.1:53680)
0927 16:28:17.864902 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.864918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.865053 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.865362 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.865365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.865372 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:17.865480 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.865485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.866196 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.866201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.866868 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.866876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.867335 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.867338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.867479 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.867533 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.868158 (+ 625us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.868173 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.868245 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.868328 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260927 16:28:17.873113 17905 raft_consensus.cc:1261] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:17.873781 17982 consensus_queue.cc:1035] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:17.873895 17625 raft_consensus.cc:1261] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:17.874547 17982 consensus_queue.cc:1035] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:17.877386 17980 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: TxnStatusTablet state changed. Reason: New leader 0205afccb32440f28ef9d9348ee6f2d3. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.877497 17980 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.877503 17981 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: TxnStatusTablet state changed. Reason: New leader 0205afccb32440f28ef9d9348ee6f2d3. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.877624 17981 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.880304 17982 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.880414 17982 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:17.880627 18002 mvcc.cc:205] Tried to move back new op lower bound from 7333996535281762304 to 7333996535204143104. Current Snapshot: MvccSnapshot[applied={T|T < 7333996535281762304}]
I20260927 16:28:17.881803 17980 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.881915 17980 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P b3e64e9fd4a5438f85cc16430ac3c46f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.882413 17997 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.882601 17997 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 0205afccb32440f28ef9d9348ee6f2d3: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:17.883654 18003 mvcc.cc:205] Tried to move back new op lower bound from 7333996535281762304 to 7333996535204143104. Current Snapshot: MvccSnapshot[applied={T|T < 7333996535281762304}]
I20260927 16:28:17.884177 17981 tablet_replica.cc:439] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } } }
I20260927 16:28:17.884284 17981 tablet_replica.cc:442] T df791a67cf18410a87589c5dc57e7e66 P 2cbf8e7ea9474e8e88b6e60c592e772c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:17.894735 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.890539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.15.93.129:32935)
0927 16:28:17.890653 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.890656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.890668 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:17.890812 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.890814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.890861 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.890940 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:17.891184 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.891840 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.893062 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.893492 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.893619 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.893968 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.894339 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.894423 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.894542 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.894587 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:28:17.905331 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.901955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:44853 (local address 127.15.93.130:44633)
0927 16:28:17.902084 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.902087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.902197 (+ 110us) server_negotiation.cc:407] Connection header received
0927 16:28:17.902318 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.902320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.902359 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.902419 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:17.902679 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.903137 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.904388 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.904588 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.904678 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.904976 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.905034 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.905084 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.905196 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.905235 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:17.905418 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.901824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44633 (local address 127.15.93.129:44853)
0927 16:28:17.902112 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.902127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.902230 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.902449 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.902451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.902460 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:17.902583 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.902589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.903232 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.903236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.904283 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.904294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.904694 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.904698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.904794 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.904841 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.905110 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.905117 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.905182 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.905257 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
I20260927 16:28:17.912021 17765 raft_consensus.cc:1261] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Refusing update from remote peer b3e64e9fd4a5438f85cc16430ac3c46f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:17.912053 17905 raft_consensus.cc:1261] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Refusing update from remote peer b3e64e9fd4a5438f85cc16430ac3c46f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:17.912500 17985 consensus_queue.cc:1035] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:17.912712 17980 consensus_queue.cc:1035] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:17.934427 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.930381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37595 (local address 127.15.93.129:32935)
0927 16:28:17.930488 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:17.930492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:17.930505 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:28:17.930629 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:17.930632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:17.930689 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:17.930769 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:17.931337 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.932172 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.933313 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.933601 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.933707 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.934002 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.934074 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.934167 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:17.934251 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 16:28:17.934300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:28:17.934448 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:17.930102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32935 (local address 127.15.93.130:37595)
0927 16:28:17.930400 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:17.930416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:17.930509 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:17.930779 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:17.930782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:17.931131 (+ 349us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:17.931225 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.931230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.932266 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.932272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.933176 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:17.933185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:17.933705 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:17.933709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:17.933806 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:17.933854 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:17.934172 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:17.934181 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:28:17.934244 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:28:17.934313 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260927 16:28:17.962263 17466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:17.962446 17466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:17.965041 17466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Sending DeleteTablet for 3 replicas of tablet b64838a15fa046939aff687083a9b927
I20260927 16:28:17.965766 17885 tablet_service.cc:1514] Processing DeleteTablet for tablet b64838a15fa046939aff687083a9b927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:17 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35976
I20260927 16:28:17.965863 17605 tablet_service.cc:1514] Processing DeleteTablet for tablet b64838a15fa046939aff687083a9b927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:17 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260927 16:28:17.966305 18023 tablet_replica.cc:330] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: stopping tablet replica
I20260927 16:28:17.966655 18023 raft_consensus.cc:2229] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:17.967010 18023 raft_consensus.cc:2258] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:17.967180 17466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:17.967566 17466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:17.968415 18023 ts_tablet_manager.cc:1905] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:17.971385 18023 ts_tablet_manager.cc:1918] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:17.971473 18023 log.cc:1196] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/b64838a15fa046939aff687083a9b927
I20260927 16:28:17.971763 18023 ts_tablet_manager.cc:1939] T b64838a15fa046939aff687083a9b927 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting consensus metadata
I20260927 16:28:17.972648 17605 tablet_service.cc:1467] Processing CreateTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (DEFAULT_TABLE table=test-table [id=397f5d8271e84278bb01a68bb2050b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.972827 17454 catalog_manager.cc:4841] TS b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935): tablet b64838a15fa046939aff687083a9b927 (table test-table [id=5cbafe759c0c42708199e5c44e9389d4]) successfully deleted
I20260927 16:28:17.972885 17605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37c361d7fc4ff89c6d22bcc99e1c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.973812 17745 tablet_service.cc:1514] Processing DeleteTablet for tablet b64838a15fa046939aff687083a9b927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:17 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260927 16:28:17.974336 17744 tablet_service.cc:1467] Processing CreateTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (DEFAULT_TABLE table=test-table [id=397f5d8271e84278bb01a68bb2050b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.974593 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37c361d7fc4ff89c6d22bcc99e1c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.975112 17974 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Bootstrap starting.
I20260927 16:28:17.976022 17974 tablet_bootstrap.cc:654] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.976830 17976 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Bootstrap starting.
I20260927 16:28:17.977308 18022 tablet_replica.cc:330] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: stopping tablet replica
I20260927 16:28:17.977468 17885 tablet_service.cc:1467] Processing CreateTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (DEFAULT_TABLE table=test-table [id=397f5d8271e84278bb01a68bb2050b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:17.977694 17976 tablet_bootstrap.cc:654] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.977743 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb37c361d7fc4ff89c6d22bcc99e1c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:17.977854 18022 raft_consensus.cc:2229] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:17.978152 18022 raft_consensus.cc:2258] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:17.979254 17976 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: No bootstrap required, opened a new log
I20260927 16:28:17.979344 17976 ts_tablet_manager.cc:1397] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:28:17.979805 17976 raft_consensus.cc:348] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.979936 17976 raft_consensus.cc:374] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.979974 17976 raft_consensus.cc:729] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.980104 17976 consensus_queue.cc:260] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.980362 17976 ts_tablet_manager.cc:1428] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:17.981902 17975 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Bootstrap starting.
I20260927 16:28:17.982792 17975 tablet_bootstrap.cc:654] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:17.983331 18022 ts_tablet_manager.cc:1905] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:17.984261 17974 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: No bootstrap required, opened a new log
I20260927 16:28:17.984356 17974 ts_tablet_manager.cc:1397] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260927 16:28:17.984798 17974 raft_consensus.cc:348] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.984925 17974 raft_consensus.cc:374] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.984964 17974 raft_consensus.cc:729] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.985086 17974 consensus_queue.cc:260] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.985348 17974 ts_tablet_manager.cc:1428] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:17.987260 18024 tablet_replica.cc:330] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: stopping tablet replica
I20260927 16:28:17.987679 18024 raft_consensus.cc:2229] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:17.987762 18022 ts_tablet_manager.cc:1918] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:17.987855 18024 raft_consensus.cc:2258] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:17.987884 18022 log.cc:1196] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/b64838a15fa046939aff687083a9b927
I20260927 16:28:17.988737 18022 ts_tablet_manager.cc:1939] T b64838a15fa046939aff687083a9b927 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting consensus metadata
I20260927 16:28:17.989168 18024 ts_tablet_manager.cc:1905] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:17.990015 17453 catalog_manager.cc:4841] TS 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459): tablet b64838a15fa046939aff687083a9b927 (table test-table [id=5cbafe759c0c42708199e5c44e9389d4]) successfully deleted
I20260927 16:28:17.992050 17975 tablet_bootstrap.cc:492] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: No bootstrap required, opened a new log
I20260927 16:28:17.992137 17975 ts_tablet_manager.cc:1397] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260927 16:28:17.992554 17975 raft_consensus.cc:348] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.992668 17975 raft_consensus.cc:374] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:17.992699 17975 raft_consensus.cc:729] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf8e7ea9474e8e88b6e60c592e772c, State: Initialized, Role: FOLLOWER
I20260927 16:28:17.992818 17975 consensus_queue.cc:260] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:17.993076 17975 ts_tablet_manager.cc:1428] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:17.993929 18024 ts_tablet_manager.cc:1918] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:17.994024 18024 log.cc:1196] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/b64838a15fa046939aff687083a9b927
I20260927 16:28:17.994326 18024 ts_tablet_manager.cc:1939] T b64838a15fa046939aff687083a9b927 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting consensus metadata
I20260927 16:28:17.995391 17452 catalog_manager.cc:4841] TS 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633): tablet b64838a15fa046939aff687083a9b927 (table test-table [id=5cbafe759c0c42708199e5c44e9389d4]) successfully deleted
I20260927 16:28:18.035996 17982 raft_consensus.cc:482] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:18.036151 17982 raft_consensus.cc:504] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:18.036566 17982 leader_election.cc:288] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:18.036936 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37c361d7fc4ff89c6d22bcc99e1c07" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" is_pre_election: true
I20260927 16:28:18.036937 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37c361d7fc4ff89c6d22bcc99e1c07" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" is_pre_election: true
I20260927 16:28:18.037086 17625 raft_consensus.cc:2454] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:18.037086 17904 raft_consensus.cc:2454] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:18.037382 17697 leader_election.cc:302] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c; no voters:
I20260927 16:28:18.037537 17982 raft_consensus.cc:2781] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:18.037597 17982 raft_consensus.cc:482] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:18.037678 17982 raft_consensus.cc:3037] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:18.038596 17982 raft_consensus.cc:504] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:18.038928 17982 leader_election.cc:288] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Requested vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:18.039252 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37c361d7fc4ff89c6d22bcc99e1c07" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
I20260927 16:28:18.039255 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb37c361d7fc4ff89c6d22bcc99e1c07" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
I20260927 16:28:18.039384 17625 raft_consensus.cc:3037] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:18.039366 17904 raft_consensus.cc:3037] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:18.040459 17904 raft_consensus.cc:2454] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:18.040459 17625 raft_consensus.cc:2454] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:18.040745 17698 leader_election.cc:302] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, b3e64e9fd4a5438f85cc16430ac3c46f; no voters:
I20260927 16:28:18.040982 17982 raft_consensus.cc:2781] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:18.041075 17982 raft_consensus.cc:686] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Running, Role: LEADER
I20260927 16:28:18.041215 17982 consensus_queue.cc:237] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:18.042770 17466 catalog_manager.cc:5495] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:18.072950 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:18.069641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.15.93.130:44633)
0927 16:28:18.069761 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:18.069765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:18.069780 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:18.069892 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:18.069895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:18.069947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:18.070039 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:18.070236 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.071020 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.071879 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.072230 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.072332 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:18.072524 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:18.072604 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:18.072674 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:18.072749 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 16:28:18.072805 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:28:18.081882 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:18.078713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:36845 (local address 127.15.93.130:44633)
0927 16:28:18.078816 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:18.078819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:18.078855 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:28:18.078979 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:18.078981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:18.079047 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:18.079127 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:18.079414 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.079886 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.080964 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.081158 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.081230 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:18.081483 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:18.081540 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:18.081590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:18.081724 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 16:28:18.081778 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":38}
I20260927 16:28:18.081992 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:18.078635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44633 (local address 127.15.93.130:36845)
0927 16:28:18.078801 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:18.078814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:18.078897 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:18.079148 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:18.079153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:18.079163 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:18.079280 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:18.079285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.079995 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.079998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:18.080850 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:18.080861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:18.081267 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:18.081271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:18.081351 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:18.081381 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:18.081637 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:18.081643 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:28:18.081715 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:28:18.081829 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":20}
I20260927 16:28:18.087249 17625 raft_consensus.cc:1261] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:18.087339 17904 raft_consensus.cc:1261] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:18.087699 18014 consensus_queue.cc:1035] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:18.087867 17982 consensus_queue.cc:1035] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:28:18.102895 18002 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:28:19.663767 17467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:19.663934 17467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:19.666483 17467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Sending DeleteTablet for 3 replicas of tablet fb37c361d7fc4ff89c6d22bcc99e1c07
I20260927 16:28:19.667024 17605 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260927 16:28:19.667112 17885 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35976
I20260927 16:28:19.667135 17744 tablet_service.cc:1514] Processing DeleteTablet for tablet fb37c361d7fc4ff89c6d22bcc99e1c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260927 16:28:19.667438 18052 tablet_replica.cc:330] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: stopping tablet replica
I20260927 16:28:19.667559 18053 tablet_replica.cc:330] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: stopping tablet replica
I20260927 16:28:19.667567 18052 raft_consensus.cc:2229] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:19.667706 18052 raft_consensus.cc:2258] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.667758 18053 raft_consensus.cc:2229] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:19.668066 18053 raft_consensus.cc:2258] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.668450 18052 ts_tablet_manager.cc:1905] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:19.668678 18053 ts_tablet_manager.cc:1905] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:19.670689 18051 tablet_replica.cc:330] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: stopping tablet replica
I20260927 16:28:19.670816 18051 raft_consensus.cc:2229] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:19.670921 18051 raft_consensus.cc:2258] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.671561 18051 ts_tablet_manager.cc:1905] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:19.671566 18053 ts_tablet_manager.cc:1918] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:28:19.671651 18053 log.cc:1196] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/fb37c361d7fc4ff89c6d22bcc99e1c07
I20260927 16:28:19.671680 18052 ts_tablet_manager.cc:1918] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:28:19.671756 18052 log.cc:1196] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/fb37c361d7fc4ff89c6d22bcc99e1c07
I20260927 16:28:19.671969 18053 ts_tablet_manager.cc:1939] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 0205afccb32440f28ef9d9348ee6f2d3: Deleting consensus metadata
I20260927 16:28:19.672072 18052 ts_tablet_manager.cc:1939] T fb37c361d7fc4ff89c6d22bcc99e1c07 P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting consensus metadata
I20260927 16:28:19.672566 17452 catalog_manager.cc:4841] TS 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633): tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (table test-table [id=397f5d8271e84278bb01a68bb2050b8f]) successfully deleted
I20260927 16:28:19.672730 17453 catalog_manager.cc:4841] TS 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459): tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (table test-table [id=397f5d8271e84278bb01a68bb2050b8f]) successfully deleted
I20260927 16:28:19.674068 18051 ts_tablet_manager.cc:1918] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:28:19.674170 18051 log.cc:1196] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/fb37c361d7fc4ff89c6d22bcc99e1c07
I20260927 16:28:19.674400 18051 ts_tablet_manager.cc:1939] T fb37c361d7fc4ff89c6d22bcc99e1c07 P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting consensus metadata
I20260927 16:28:19.674983 17454 catalog_manager.cc:4841] TS b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935): tablet fb37c361d7fc4ff89c6d22bcc99e1c07 (table test-table [id=397f5d8271e84278bb01a68bb2050b8f]) successfully deleted
Sep 27 16:28:19 dist-test-slave-grzf krb5kdc[17423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:28:19 dist-test-slave-grzf krb5kdc[17423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:19.697381 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:19.686057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.15.93.190:39705)
0927 16:28:19.686312 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:19.686320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:19.686414 (+ 94us) server_negotiation.cc:407] Connection header received
0927 16:28:19.686570 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:19.686579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:19.686622 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:19.686732 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:19.687406 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:19.687967 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:19.688790 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:19.689022 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:19.689128 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:19.694374 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:19.694377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:19.694379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:19.694400 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:19.695760 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:19.696176 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:19.696180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:19.696182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:19.696267 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:19.696535 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:19.696540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:19.696541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:19.696717 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:19.696912 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:19.697089 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 16:28:19.697212 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260927 16:28:19.699749 17467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:19.700177 17467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:19.704578 17605 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d319e31124fe193cde6394fe1d80a (DEFAULT_TABLE table=test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.704593 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d319e31124fe193cde6394fe1d80a (DEFAULT_TABLE table=test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.704685 17885 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0d319e31124fe193cde6394fe1d80a (DEFAULT_TABLE table=test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.704869 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d319e31124fe193cde6394fe1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.704874 17605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d319e31124fe193cde6394fe1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.704906 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0d319e31124fe193cde6394fe1d80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.706733 18062 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Bootstrap starting.
I20260927 16:28:19.707525 18064 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Bootstrap starting.
I20260927 16:28:19.707549 18063 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Bootstrap starting.
I20260927 16:28:19.707646 18062 tablet_bootstrap.cc:654] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.708379 18063 tablet_bootstrap.cc:654] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.708379 18064 tablet_bootstrap.cc:654] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.708709 18062 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: No bootstrap required, opened a new log
I20260927 16:28:19.708812 18062 ts_tablet_manager.cc:1397] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:19.709280 18062 raft_consensus.cc:348] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.709398 18062 raft_consensus.cc:374] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.709435 18062 raft_consensus.cc:729] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.709607 18062 consensus_queue.cc:260] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.709939 18063 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: No bootstrap required, opened a new log
I20260927 16:28:19.710019 18063 ts_tablet_manager.cc:1397] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:28:19.710021 18062 ts_tablet_manager.cc:1428] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:19.710121 18064 tablet_bootstrap.cc:492] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: No bootstrap required, opened a new log
I20260927 16:28:19.710186 18064 ts_tablet_manager.cc:1397] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:19.710431 18063 raft_consensus.cc:348] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.710552 18063 raft_consensus.cc:374] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.710604 18063 raft_consensus.cc:729] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.710577 18064 raft_consensus.cc:348] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.710695 18064 raft_consensus.cc:374] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.710726 18064 raft_consensus.cc:729] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf8e7ea9474e8e88b6e60c592e772c, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.710734 18063 consensus_queue.cc:260] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.710842 18064 consensus_queue.cc:260] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.711004 18063 ts_tablet_manager.cc:1428] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:19.711179 18064 ts_tablet_manager.cc:1428] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:19.869800 18039 raft_consensus.cc:482] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:19.869932 18039 raft_consensus.cc:504] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.870389 18039 leader_election.cc:288] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:19.870814 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d319e31124fe193cde6394fe1d80a" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" is_pre_election: true
I20260927 16:28:19.870841 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d319e31124fe193cde6394fe1d80a" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" is_pre_election: true
I20260927 16:28:19.870944 17904 raft_consensus.cc:2454] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:19.871044 17625 raft_consensus.cc:2454] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:19.871322 17697 leader_election.cc:302] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c; no voters:
I20260927 16:28:19.871527 18039 raft_consensus.cc:2781] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:19.871592 18039 raft_consensus.cc:482] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:19.871622 18039 raft_consensus.cc:3037] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:19.872619 18039 raft_consensus.cc:504] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.872937 18039 leader_election.cc:288] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Requested vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:19.873242 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d319e31124fe193cde6394fe1d80a" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
I20260927 16:28:19.873287 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0d319e31124fe193cde6394fe1d80a" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
I20260927 16:28:19.873397 17904 raft_consensus.cc:3037] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:19.873394 17625 raft_consensus.cc:3037] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:19.874284 17904 raft_consensus.cc:2454] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:19.874300 17625 raft_consensus.cc:2454] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:19.874580 17697 leader_election.cc:302] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c; no voters:
I20260927 16:28:19.874732 18039 raft_consensus.cc:2781] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:19.874838 18039 raft_consensus.cc:686] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Running, Role: LEADER
I20260927 16:28:19.874977 18039 consensus_queue.cc:237] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.876721 17466 catalog_manager.cc:5495] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:19.902952 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:19.900090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.15.93.130:44633)
0927 16:28:19.900343 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:19.900350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:19.900361 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:19.900397 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:19.900399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:19.900438 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:19.900508 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:19.900729 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:19.901252 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:19.902040 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:19.902260 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:19.902337 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:19.902532 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:19.902593 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:19.902648 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:19.902729 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 16:28:19.902769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260927 16:28:19.908629 17904 raft_consensus.cc:1261] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:19.908680 17625 raft_consensus.cc:1261] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:19.908999 18039 consensus_queue.cc:1035] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:19.909152 18034 consensus_queue.cc:1035] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:19.914563 18035 mvcc.cc:205] Tried to move back new op lower bound from 7333996543622893568 to 7333996543489613824. Current Snapshot: MvccSnapshot[applied={T|T < 7333996543622893568}]
I20260927 16:28:19.935472 17466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:19.935603 17466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:19.937319 17466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Sending DeleteTablet for 3 replicas of tablet 3b0d319e31124fe193cde6394fe1d80a
I20260927 16:28:19.937801 17605 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d319e31124fe193cde6394fe1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260927 16:28:19.938002 18051 tablet_replica.cc:330] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: stopping tablet replica
I20260927 16:28:19.938143 18051 raft_consensus.cc:2229] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:19.938197 17885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d319e31124fe193cde6394fe1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35976
I20260927 16:28:19.938253 18051 raft_consensus.cc:2258] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.938380 18052 tablet_replica.cc:330] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: stopping tablet replica
I20260927 16:28:19.938444 17744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0d319e31124fe193cde6394fe1d80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:19 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260927 16:28:19.938483 18052 raft_consensus.cc:2229] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:19.938570 18052 raft_consensus.cc:2258] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.938606 18053 tablet_replica.cc:330] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: stopping tablet replica
I20260927 16:28:19.938727 18053 raft_consensus.cc:2229] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:19.938881 18053 raft_consensus.cc:2258] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:19.939199 18052 ts_tablet_manager.cc:1905] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:19.939148 17466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 16:28:19.939246 18051 ts_tablet_manager.cc:1905] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:28:19.939550 17466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:19.940596 18053 ts_tablet_manager.cc:1905] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:19.941656 18052 ts_tablet_manager.cc:1918] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:19.941736 18052 log.cc:1196] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/3b0d319e31124fe193cde6394fe1d80a
I20260927 16:28:19.941764 18051 ts_tablet_manager.cc:1918] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:19.941838 18051 log.cc:1196] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/3b0d319e31124fe193cde6394fe1d80a
I20260927 16:28:19.942004 18052 ts_tablet_manager.cc:1939] T 3b0d319e31124fe193cde6394fe1d80a P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting consensus metadata
I20260927 16:28:19.942098 18051 ts_tablet_manager.cc:1939] T 3b0d319e31124fe193cde6394fe1d80a P b3e64e9fd4a5438f85cc16430ac3c46f: Deleting consensus metadata
I20260927 16:28:19.942761 17454 catalog_manager.cc:4841] TS b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935): tablet 3b0d319e31124fe193cde6394fe1d80a (table test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]) successfully deleted
I20260927 16:28:19.943349 18053 ts_tablet_manager.cc:1918] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:19.943459 18053 log.cc:1196] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/3b0d319e31124fe193cde6394fe1d80a
I20260927 16:28:19.943722 18053 ts_tablet_manager.cc:1939] T 3b0d319e31124fe193cde6394fe1d80a P 0205afccb32440f28ef9d9348ee6f2d3: Deleting consensus metadata
I20260927 16:28:19.944172 17605 tablet_service.cc:1467] Processing CreateTablet for tablet 26df3fcca706432f93d760d52c37c1ff (DEFAULT_TABLE table=test-table [id=174ad142ed7b4c60b7bf93ae0c5885fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.944425 17605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26df3fcca706432f93d760d52c37c1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.944533 17885 tablet_service.cc:1467] Processing CreateTablet for tablet 26df3fcca706432f93d760d52c37c1ff (DEFAULT_TABLE table=test-table [id=174ad142ed7b4c60b7bf93ae0c5885fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.944753 17885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26df3fcca706432f93d760d52c37c1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.944783 17452 catalog_manager.cc:4841] TS 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633): tablet 3b0d319e31124fe193cde6394fe1d80a (table test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]) successfully deleted
I20260927 16:28:19.946511 18062 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: Bootstrap starting.
I20260927 16:28:19.946789 18064 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Bootstrap starting.
I20260927 16:28:19.946851 17453 catalog_manager.cc:4841] TS 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459): tablet 3b0d319e31124fe193cde6394fe1d80a (table test-table [id=7f0487c5915d463e8ac4ef0731c82cbe]) successfully deleted
I20260927 16:28:19.947345 18062 tablet_bootstrap.cc:654] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.947319 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 26df3fcca706432f93d760d52c37c1ff (DEFAULT_TABLE table=test-table [id=174ad142ed7b4c60b7bf93ae0c5885fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:19.947553 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26df3fcca706432f93d760d52c37c1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:19.947688 18064 tablet_bootstrap.cc:654] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.948822 18064 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: No bootstrap required, opened a new log
I20260927 16:28:19.948908 18064 ts_tablet_manager.cc:1397] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:19.949309 18064 raft_consensus.cc:348] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.949437 18064 raft_consensus.cc:374] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.949471 18064 raft_consensus.cc:729] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf8e7ea9474e8e88b6e60c592e772c, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.949591 18064 consensus_queue.cc:260] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.949653 18063 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Bootstrap starting.
I20260927 16:28:19.949898 18064 ts_tablet_manager.cc:1428] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:19.950245 18062 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: No bootstrap required, opened a new log
I20260927 16:28:19.950317 18062 ts_tablet_manager.cc:1397] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 16:28:19.950429 18063 tablet_bootstrap.cc:654] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:19.950708 18062 raft_consensus.cc:348] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.950824 18062 raft_consensus.cc:374] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.950861 18062 raft_consensus.cc:729] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e64e9fd4a5438f85cc16430ac3c46f, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.950980 18062 consensus_queue.cc:260] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.951267 18062 ts_tablet_manager.cc:1428] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:19.951509 18063 tablet_bootstrap.cc:492] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: No bootstrap required, opened a new log
I20260927 16:28:19.951582 18063 ts_tablet_manager.cc:1397] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:28:19.951942 18063 raft_consensus.cc:348] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.952018 18063 raft_consensus.cc:374] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:19.952041 18063 raft_consensus.cc:729] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Initialized, Role: FOLLOWER
I20260927 16:28:19.952160 18063 consensus_queue.cc:260] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:19.952417 18063 ts_tablet_manager.cc:1428] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:20.278144 17982 raft_consensus.cc:482] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:20.278288 17982 raft_consensus.cc:504] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:20.278720 17982 leader_election.cc:288] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:20.279188 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26df3fcca706432f93d760d52c37c1ff" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" is_pre_election: true
I20260927 16:28:20.279188 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26df3fcca706432f93d760d52c37c1ff" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" is_pre_election: true
I20260927 16:28:20.279337 17625 raft_consensus.cc:2454] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:20.279337 17904 raft_consensus.cc:2454] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 0.
I20260927 16:28:20.279695 17698 leader_election.cc:302] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, 2cbf8e7ea9474e8e88b6e60c592e772c, b3e64e9fd4a5438f85cc16430ac3c46f; no voters:
I20260927 16:28:20.279870 17982 raft_consensus.cc:2781] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:20.279924 17982 raft_consensus.cc:482] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:20.279968 17982 raft_consensus.cc:3037] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:20.280917 17982 raft_consensus.cc:504] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:20.281237 17982 leader_election.cc:288] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Requested vote from peers b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935), 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459)
I20260927 16:28:20.281514 17625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26df3fcca706432f93d760d52c37c1ff" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f"
I20260927 16:28:20.281597 17904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26df3fcca706432f93d760d52c37c1ff" candidate_uuid: "0205afccb32440f28ef9d9348ee6f2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c"
I20260927 16:28:20.281669 17625 raft_consensus.cc:3037] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:20.281749 17904 raft_consensus.cc:3037] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:20.282681 17625 raft_consensus.cc:2454] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:20.282729 17904 raft_consensus.cc:2454] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0205afccb32440f28ef9d9348ee6f2d3 in term 1.
I20260927 16:28:20.282985 17698 leader_election.cc:302] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0205afccb32440f28ef9d9348ee6f2d3, b3e64e9fd4a5438f85cc16430ac3c46f; no voters:
I20260927 16:28:20.283151 17982 raft_consensus.cc:2781] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:20.283239 17982 raft_consensus.cc:686] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 0205afccb32440f28ef9d9348ee6f2d3, State: Running, Role: LEADER
I20260927 16:28:20.283398 17982 consensus_queue.cc:237] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } }
I20260927 16:28:20.285025 17467 catalog_manager.cc:5495] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "0205afccb32440f28ef9d9348ee6f2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0205afccb32440f28ef9d9348ee6f2d3" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:20.321801 17625 raft_consensus.cc:1261] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:20.322127 17904 raft_consensus.cc:1261] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Refusing update from remote peer 0205afccb32440f28ef9d9348ee6f2d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:20.322197 17982 consensus_queue.cc:1035] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e64e9fd4a5438f85cc16430ac3c46f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:20.322461 18039 consensus_queue.cc:1035] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf8e7ea9474e8e88b6e60c592e772c" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:28:20.333945 18032 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 16:28:21.884373 17466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:21.884526 17466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:21.886850 17466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e740df81e0344d9c9306b32b6f2a7a5b: Sending DeleteTablet for 3 replicas of tablet 26df3fcca706432f93d760d52c37c1ff
I20260927 16:28:21.887418 17744 tablet_service.cc:1514] Processing DeleteTablet for tablet 26df3fcca706432f93d760d52c37c1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:21 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260927 16:28:21.887437 17605 tablet_service.cc:1514] Processing DeleteTablet for tablet 26df3fcca706432f93d760d52c37c1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:21 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260927 16:28:21.887532 17885 tablet_service.cc:1514] Processing DeleteTablet for tablet 26df3fcca706432f93d760d52c37c1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:21 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35976
I20260927 16:28:21.887809 18103 tablet_replica.cc:330] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f: stopping tablet replica
I20260927 16:28:21.887856 18104 tablet_replica.cc:330] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: stopping tablet replica
I20260927 16:28:21.887867 18102 tablet_replica.cc:330] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: stopping tablet replica
I20260927 16:28:21.887989 18104 raft_consensus.cc:2229] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:21.887989 18102 raft_consensus.cc:2229] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:21.888106 18104 raft_consensus.cc:2258] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:21.888103 18103 raft_consensus.cc:2229] T 26df3fcca706432f93d760d52c37c1ff P b3e64e9fd4a5438f85cc16430ac3c46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:21.888175 18102 raft_consensus.cc:2258] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:21.888190 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17540
I20260927 16:28:21.889369 18102 ts_tablet_manager.cc:1905] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:21.889673 18104 ts_tablet_manager.cc:1905] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:21.891830 18104 ts_tablet_manager.cc:1918] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:28:21.891922 18104 log.cc:1196] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/26df3fcca706432f93d760d52c37c1ff
I20260927 16:28:21.892228 18104 ts_tablet_manager.cc:1939] T 26df3fcca706432f93d760d52c37c1ff P 2cbf8e7ea9474e8e88b6e60c592e772c: Deleting consensus metadata
I20260927 16:28:21.892606 18102 ts_tablet_manager.cc:1918] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 16:28:21.892688 18102 log.cc:1196] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/26df3fcca706432f93d760d52c37c1ff
I20260927 16:28:21.892856 17453 catalog_manager.cc:4841] TS 2cbf8e7ea9474e8e88b6e60c592e772c (127.15.93.131:38459): tablet 26df3fcca706432f93d760d52c37c1ff (table test-table [id=174ad142ed7b4c60b7bf93ae0c5885fb]) successfully deleted
I20260927 16:28:21.892975 18102 ts_tablet_manager.cc:1939] T 26df3fcca706432f93d760d52c37c1ff P 0205afccb32440f28ef9d9348ee6f2d3: Deleting consensus metadata
I20260927 16:28:21.895555 17452 catalog_manager.cc:4841] TS 0205afccb32440f28ef9d9348ee6f2d3 (127.15.93.130:44633): tablet 26df3fcca706432f93d760d52c37c1ff (table test-table [id=174ad142ed7b4c60b7bf93ae0c5885fb]) successfully deleted
I20260927 16:28:21.897296 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17680
W20260927 16:28:21.897262 17454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:32935 (error 108)
I20260927 16:28:21.899108 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:21.898737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32935 (local address 127.0.0.1:33536)
0927 16:28:21.898968 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:21.899031 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20260927 16:28:21.899236 17454 catalog_manager.cc:4584] TS b3e64e9fd4a5438f85cc16430ac3c46f (127.15.93.129:32935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26df3fcca706432f93d760d52c37c1ff: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32935: connect: Connection refused (error 111)
I20260927 16:28:21.904407 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17820
I20260927 16:28:21.910176 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 17436
2026-09-27T16:28:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6416 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:21 dist-test-slave-grzf krb5kdc[18109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:21 dist-test-slave-grzf krb5kdc[18109](info): set up 2 sockets
Sep 27 16:28:21 dist-test-slave-grzf krb5kdc[18109](info): commencing operation
krb5kdc: starting...
W20260927 16:28:22.955612 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:23.069449 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43127
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:43127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:23.178718 18122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:23.179018 18122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:23.179097 18122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:23.183087 18122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:23.183171 18122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:23.183210 18122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:23.183269 18122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:23.183306 18122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:23.188509 18122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:43127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:23.189508 18122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:23.190502 18122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:23.198390 18130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.198422 18122 server_base.cc:1034] running on GCE node
W20260927 16:28:23.198374 18127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.198370 18128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.199158 18122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:23.200229 18122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:23.201418 18122 hybrid_clock.cc:648] HybridClock initialized: now 1790526503201408 us; error 28 us; skew 500 ppm
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:23.204216 18122 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:23.205051 18122 webserver.cc:460] Webserver started at http://127.15.93.190:38427/ using document root <none> and password file <none>
I20260927 16:28:23.205531 18122 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:23.205585 18122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:23.205746 18122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:23.207705 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "a5b0364e4a80487e937bbcdb7e211e2b"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "fdbbaaff0852328dd59c88e305039e77"
server_key_iv: "cf5afa575e650e8beb207699421502cf"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.208185 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "a5b0364e4a80487e937bbcdb7e211e2b"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "fdbbaaff0852328dd59c88e305039e77"
server_key_iv: "cf5afa575e650e8beb207699421502cf"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.211587 18122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:28:23.213744 18137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.215042 18122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:23.215176 18122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "a5b0364e4a80487e937bbcdb7e211e2b"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "fdbbaaff0852328dd59c88e305039e77"
server_key_iv: "cf5afa575e650e8beb207699421502cf"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.215281 18122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:23.238914 18122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:23.241884 18122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:23.242100 18122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:23.249027 18122 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:43127
I20260927 16:28:23.249095 18189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:43127 every 8 connection(s)
I20260927 16:28:23.249981 18122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:23.252650 18190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:23.255607 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18122
I20260927 16:28:23.255733 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:23.255978 15734 external_mini_cluster.cc:1428] Setting key d79180d5227818a7ffb6a2c92f29b45d
I20260927 16:28:23.258494 18190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: Bootstrap starting.
I20260927 16:28:23.261070 18190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:23.261845 18190 log.cc:824] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:23.263942 18190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: No bootstrap required, opened a new log
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:23.267937 18190 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } }
I20260927 16:28:23.268211 18190 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:23.268277 18190 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b0364e4a80487e937bbcdb7e211e2b, State: Initialized, Role: FOLLOWER
I20260927 16:28:23.268749 18190 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } }
I20260927 16:28:23.268908 18190 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:23.268986 18190 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:23.269094 18190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:23.269932 18190 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } }
I20260927 16:28:23.270311 18190 leader_election.cc:302] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5b0364e4a80487e937bbcdb7e211e2b; no voters:
I20260927 16:28:23.270690 18190 leader_election.cc:288] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:23.270781 18195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:23.270978 18195 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [term 1 LEADER]: Becoming Leader. State: Replica: a5b0364e4a80487e937bbcdb7e211e2b, State: Running, Role: LEADER
I20260927 16:28:23.271322 18195 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } }
I20260927 16:28:23.271538 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.257269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.15.93.190:43127)
0927 16:28:23.257682 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:23.257704 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:23.257742 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:28:23.258485 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:23.258510 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:23.258919 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:23.259260 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:23.259975 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.261441 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.262348 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.262949 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.263105 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.265692 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:23.265699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:23.265714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:23.265747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:23.268290 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.268754 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.268762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.268770 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.268911 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.269272 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.269276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.269278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.269698 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:23.269891 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:23.270176 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 16:28:23.270489 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260927 16:28:23.271981 18190 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:23.273223 18196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } } }
I20260927 16:28:23.273340 18196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:23.274199 18197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5b0364e4a80487e937bbcdb7e211e2b. Latest consensus state: current_term: 1 leader_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5b0364e4a80487e937bbcdb7e211e2b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43127 } } }
I20260927 16:28:23.274291 18197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:23.276913 18211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:23.276988 18211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:23.277060 18205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:23.277742 18205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:23.283236 18205 catalog_manager.cc:1347] Generated new cluster ID: d6822703d3b34d50ad84ee730dbba54e
I20260927 16:28:23.283303 18205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:23.297472 18205 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:23.298767 18205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:23.309602 18205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5b0364e4a80487e937bbcdb7e211e2b: Generated new TSK 0
I20260927 16:28:23.310448 18205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:23.340540 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:23.450104 18216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:23.450353 18216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:23.450395 18216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:23.454285 18216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:23.454339 18216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:23.454396 18216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:23.460054 18216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:23.461045 18216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:23.462086 18216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:23.471714 18222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.471874 18216 server_base.cc:1034] running on GCE node
W20260927 16:28:23.471696 18224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.471693 18221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.472450 18216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:23.473165 18216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:23.474349 18216 hybrid_clock.cc:648] HybridClock initialized: now 1790526503474334 us; error 27 us; skew 500 ppm
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:23.477208 18216 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:23.478039 18216 webserver.cc:460] Webserver started at http://127.15.93.129:37267/ using document root <none> and password file <none>
I20260927 16:28:23.478585 18216 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:23.478673 18216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:23.478906 18216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:23.480930 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "42eee3b7b2dd44078528d42620f731fb"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "e81434308da6e2bd394cb8c4c9a4b966"
server_key_iv: "2f3513270f9066cf0aeda591cd616975"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.481485 18216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "42eee3b7b2dd44078528d42620f731fb"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "e81434308da6e2bd394cb8c4c9a4b966"
server_key_iv: "2f3513270f9066cf0aeda591cd616975"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.485440 18216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 16:28:23.487713 18231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.488945 18216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:23.489080 18216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "42eee3b7b2dd44078528d42620f731fb"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "e81434308da6e2bd394cb8c4c9a4b966"
server_key_iv: "2f3513270f9066cf0aeda591cd616975"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.489190 18216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:23.509166 18216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:23.512020 18216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:23.512217 18216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:23.512782 18216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:23.513569 18216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:23.513644 18216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.513715 18216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:23.513748 18216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.522634 18216 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:43115
I20260927 16:28:23.522651 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:43115 every 8 connection(s)
I20260927 16:28:23.523540 18216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:23.527148 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18216
I20260927 16:28:23.527271 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:23.527613 15734 external_mini_cluster.cc:1428] Setting key c23e1e1aa78cc897136692eee38e934c
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:23.537679 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.525199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:49981 (local address 127.15.93.190:43127)
0927 16:28:23.525331 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:23.525334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:23.526286 (+ 952us) server_negotiation.cc:407] Connection header received
0927 16:28:23.527290 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:23.527296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:23.527353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:23.527476 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:23.528959 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.529455 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.530961 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.531355 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.531500 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.534176 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:23.534184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:23.534186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:23.534207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:23.535663 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.536169 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.536173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.536175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.536251 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.536616 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.536622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.536623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.536876 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:23.536953 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:23.537387 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 16:28:23.537482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:23.538434 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.525542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43127 (local address 127.15.93.129:49981)
0927 16:28:23.526149 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:23.526187 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:23.527009 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:23.527677 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:23.527685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:23.528153 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:23.528662 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.528675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.529581 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.529588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.530841 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.530851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.531536 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.531543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.531774 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.532368 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:23.532393 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:23.534013 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:23.535784 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.535789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.535803 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.536050 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.536345 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.536348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.536349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.536481 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.536976 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:23.536980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:23.537193 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 16:28:23.537339 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 16:28:23.537603 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:23.539691 18345 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43127
I20260927 16:28:23.539912 18345 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:23.540351 18345 heartbeater.cc:511] Master 127.15.93.190:43127 requested a full tablet report, sending...
I20260927 16:28:23.541956 18154 ts_manager.cc:194] Registered new tserver with Master: 42eee3b7b2dd44078528d42620f731fb (127.15.93.129:43115)
I20260927 16:28:23.543609 18154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:49981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:23.554088 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:23.661242 18350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:23.661481 18350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:23.661571 18350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:23.665570 18350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:23.665620 18350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:23.665714 18350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:23.670699 18350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:23.671730 18350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:23.672791 18350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:23.682049 18358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.682044 18355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.682042 18356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.682376 18350 server_base.cc:1034] running on GCE node
I20260927 16:28:23.682811 18350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:23.683449 18350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:23.684636 18350 hybrid_clock.cc:648] HybridClock initialized: now 1790526503684626 us; error 28 us; skew 500 ppm
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:23.687222 18350 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:23.688061 18350 webserver.cc:460] Webserver started at http://127.15.93.130:33457/ using document root <none> and password file <none>
I20260927 16:28:23.688596 18350 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:23.688691 18350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:23.688934 18350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:23.690817 18350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "43cb65ac770c46f1862bb927b1d507a9"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "0ba7d3291f380bebf734c0d989a0afca"
server_key_iv: "95f3401dccb3959ed0844451086a6b2c"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.691294 18350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "43cb65ac770c46f1862bb927b1d507a9"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "0ba7d3291f380bebf734c0d989a0afca"
server_key_iv: "95f3401dccb3959ed0844451086a6b2c"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.694864 18350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 16:28:23.697116 18365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.698240 18350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:23.698325 18350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "43cb65ac770c46f1862bb927b1d507a9"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "0ba7d3291f380bebf734c0d989a0afca"
server_key_iv: "95f3401dccb3959ed0844451086a6b2c"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.698400 18350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:23.723188 18350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:23.725961 18350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:23.726104 18350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:23.726585 18350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:23.727298 18350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:23.727342 18350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.727439 18350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:23.727468 18350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.735883 18350 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:33825
I20260927 16:28:23.735962 18478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:33825 every 8 connection(s)
I20260927 16:28:23.736783 18350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:23.739828 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18350
I20260927 16:28:23.739915 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:23.740168 15734 external_mini_cluster.cc:1428] Setting key 218df903351221c1dd1eeaf3a38a85e0
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:23.750123 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.738481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:49937 (local address 127.15.93.190:43127)
0927 16:28:23.738622 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:23.738627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:23.739464 (+ 837us) server_negotiation.cc:407] Connection header received
0927 16:28:23.740823 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:23.740827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:23.740882 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:23.740990 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:23.742105 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.742615 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.743858 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.744074 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.744181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.746694 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:23.746697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:23.746699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:23.746721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:23.748235 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.748757 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.748763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.748765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.748861 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.749180 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.749183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.749185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.749352 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:23.749429 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:23.749888 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 16:28:23.749982 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:28:23.750984 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.738889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43127 (local address 127.15.93.130:49937)
0927 16:28:23.739309 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:23.739349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:23.740616 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:23.741168 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:23.741174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:23.741508 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:23.741962 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.741974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.742747 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.742750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.743733 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.743747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.744231 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.744235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.744396 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.745067 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:23.745094 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:23.746527 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:23.748350 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.748355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.748369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.748619 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.748963 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.748965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.748966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.749090 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.749439 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:23.749444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:23.749686 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 16:28:23.749835 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:28:23.750066 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:23.752234 18479 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43127
I20260927 16:28:23.752475 18479 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:23.752995 18479 heartbeater.cc:511] Master 127.15.93.190:43127 requested a full tablet report, sending...
I20260927 16:28:23.753999 18154 ts_manager.cc:194] Registered new tserver with Master: 43cb65ac770c46f1862bb927b1d507a9 (127.15.93.130:33825)
I20260927 16:28:23.754726 18154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:49937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:23.766789 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:23.874254 18484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:23.874534 18484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:23.874614 18484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:23.878685 18484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:23.878737 18484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:23.878832 18484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:23.884095 18484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:23.885136 18484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:23.886129 18484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:23.895772 18492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.895766 18490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:23.895802 18489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:23.895922 18484 server_base.cc:1034] running on GCE node
I20260927 16:28:23.896515 18484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:23.897152 18484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:23.898360 18484 hybrid_clock.cc:648] HybridClock initialized: now 1790526503898330 us; error 41 us; skew 500 ppm
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:23.901057 18484 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:23.902097 18484 webserver.cc:460] Webserver started at http://127.15.93.131:45093/ using document root <none> and password file <none>
I20260927 16:28:23.902649 18484 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:23.902747 18484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:23.902971 18484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:23.904999 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "a1c6fc0b33ca437daf6a5888b95acd00"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "05d4d90118129a3f9fda11882c915e5f"
server_key_iv: "d4afbab75b3a2444752b4ac44fe5b22f"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.905577 18484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "a1c6fc0b33ca437daf6a5888b95acd00"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "05d4d90118129a3f9fda11882c915e5f"
server_key_iv: "d4afbab75b3a2444752b4ac44fe5b22f"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.909152 18484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:23.911669 18499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.913008 18484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:28:23.913138 18484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "a1c6fc0b33ca437daf6a5888b95acd00"
format_stamp: "Formatted at 2026-09-27 16:28:23 on dist-test-slave-grzf"
server_key: "05d4d90118129a3f9fda11882c915e5f"
server_key_iv: "d4afbab75b3a2444752b4ac44fe5b22f"
server_key_version: "encryptionkey@0"
I20260927 16:28:23.913252 18484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:23.939291 18484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:23.942114 18484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:23.942312 18484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:23.942870 18484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:23.943722 18484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:23.943794 18484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.943863 18484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:23.943897 18484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:23.952346 18484 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:32829
I20260927 16:28:23.952385 18612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:32829 every 8 connection(s)
I20260927 16:28:23.953207 18484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:28:23 dist-test-slave-grzf krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:23.962554 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18484
I20260927 16:28:23.962661 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:23.962960 15734 external_mini_cluster.cc:1428] Setting key 2ffef32b3238b015b5f03ba206bb7475
I20260927 16:28:23.966667 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.954934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:50097 (local address 127.15.93.190:43127)
0927 16:28:23.955067 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:23.955071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:23.955925 (+ 854us) server_negotiation.cc:407] Connection header received
0927 16:28:23.957202 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:23.957206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:23.957261 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:23.957359 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:23.958450 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.958971 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.960148 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.960364 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.960474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.963121 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:23.963129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:23.963131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:23.963153 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:23.965022 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.965467 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.965469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.965471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.965540 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:23.965831 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:23.965833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:23.965834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:23.965988 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:23.966058 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:23.966467 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 16:28:23.966554 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:28:23.967509 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:23.955295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43127 (local address 127.15.93.131:50097)
0927 16:28:23.955774 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:23.955814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:23.956961 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:23.957521 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:23.957528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:23.957866 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:23.958310 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.958323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.959092 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.959096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.960017 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:23.960026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:23.960509 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:23.960513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:23.960684 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:23.961214 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:23.961229 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:23.962917 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:23.965122 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.965132 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.965145 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.965383 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.965616 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:23.965618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:23.965619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:23.965754 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:23.966067 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:23.966072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:23.966281 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 16:28:23.966424 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 16:28:23.966662 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":108,"threads_started":1}
I20260927 16:28:23.968580 18613 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43127
I20260927 16:28:23.968780 18613 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:23.969180 18613 heartbeater.cc:511] Master 127.15.93.190:43127 requested a full tablet report, sending...
I20260927 16:28:23.970049 18154 ts_manager.cc:194] Registered new tserver with Master: a1c6fc0b33ca437daf6a5888b95acd00 (127.15.93.131:32829)
I20260927 16:28:23.970592 18154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:50097
I20260927 16:28:23.976789 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:28:23.988883 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:23.987483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.15.93.190:43127)
0927 16:28:23.987674 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:23.987679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:23.987838 (+ 159us) server_negotiation.cc:407] Connection header received
0927 16:28:23.987946 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:23.987948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:23.987989 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:23.988060 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:23.988719 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35222: BlockingRecv error: recv got EOF from 127.0.0.1:35222 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 16:28:23.990350 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18216
I20260927 16:28:23.995860 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18350
I20260927 16:28:24.001479 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18484
I20260927 16:28:24.006717 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18122
2026-09-27T16:28:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2096 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:24 dist-test-slave-grzf krb5kdc[18625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:24 dist-test-slave-grzf krb5kdc[18625](info): set up 2 sockets
Sep 27 16:28:24 dist-test-slave-grzf krb5kdc[18625](info): commencing operation
krb5kdc: starting...
W20260927 16:28:25.043684 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:25.162034 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40091
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:40091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:25.268605 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:25.268850 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:25.268890 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:25.272787 18638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:25.272836 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:25.272856 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:25.272872 18638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:25.272887 18638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:25.277952 18638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:40091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:25.278873 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:25.279800 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:25.287719 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:25.287756 18638 server_base.cc:1034] running on GCE node
W20260927 16:28:25.287720 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:25.287704 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:25.288446 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:25.289423 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:25.290577 18638 hybrid_clock.cc:648] HybridClock initialized: now 1790526505290547 us; error 44 us; skew 500 ppm
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526505, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:25.299841 18638 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:25.300760 18638 webserver.cc:460] Webserver started at http://127.15.93.190:41571/ using document root <none> and password file <none>
I20260927 16:28:25.301301 18638 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:25.301360 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:25.301564 18638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:25.303601 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "cf8063d8897f4e999099a675c7fb91c5"
format_stamp: "Formatted at 2026-09-27 16:28:25 on dist-test-slave-grzf"
server_key: "44ef6a3d0ee2d4f855b8f3283b051038"
server_key_iv: "4a05f4845ccbea8244de2d050961b01f"
server_key_version: "encryptionkey@0"
I20260927 16:28:25.304148 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "cf8063d8897f4e999099a675c7fb91c5"
format_stamp: "Formatted at 2026-09-27 16:28:25 on dist-test-slave-grzf"
server_key: "44ef6a3d0ee2d4f855b8f3283b051038"
server_key_iv: "4a05f4845ccbea8244de2d050961b01f"
server_key_version: "encryptionkey@0"
I20260927 16:28:25.307699 18638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:25.309988 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:25.311154 18638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:25.311280 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "cf8063d8897f4e999099a675c7fb91c5"
format_stamp: "Formatted at 2026-09-27 16:28:25 on dist-test-slave-grzf"
server_key: "44ef6a3d0ee2d4f855b8f3283b051038"
server_key_iv: "4a05f4845ccbea8244de2d050961b01f"
server_key_version: "encryptionkey@0"
I20260927 16:28:25.311414 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:25.323257 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:25.325840 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:25.326025 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:25.332849 18638 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:40091
I20260927 16:28:25.333148 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:40091 every 8 connection(s)
I20260927 16:28:25.333879 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:25.336592 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:25.337755 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18638
I20260927 16:28:25.337846 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:25.338138 15734 external_mini_cluster.cc:1428] Setting key 6ec5401724c8fed27f92d902112f3a12
I20260927 16:28:25.343461 18706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5: Bootstrap starting.
I20260927 16:28:25.346076 18706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:25.346835 18706 log.cc:824] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:25.349735 18706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5: No bootstrap required, opened a new log
I20260927 16:28:25.353369 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:25.339548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.15.93.190:40091)
0927 16:28:25.339985 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:25.340005 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:25.340048 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:28:25.340882 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:25.340905 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:25.341330 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:25.341718 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:25.342411 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:25.343854 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:25.344710 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:25.345377 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:25.345545 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:25.348157 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:25.348163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:25.348182 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:25.348218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:25.350237 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:25.350677 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:25.350683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:25.350689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:25.350824 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:25.351074 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:25.351077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:25.351078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:25.351599 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:25.351791 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:25.352055 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 16:28:25.352420 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260927 16:28:25.354339 18706 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } }
I20260927 16:28:25.354562 18706 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:25.354642 18706 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8063d8897f4e999099a675c7fb91c5, State: Initialized, Role: FOLLOWER
I20260927 16:28:25.355103 18706 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } }
I20260927 16:28:25.355232 18706 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:25.355314 18706 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:25.355480 18706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:25.356446 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:28:25.356554 18706 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } }
I20260927 16:28:25.356894 18706 leader_election.cc:302] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8063d8897f4e999099a675c7fb91c5; no voters:
I20260927 16:28:25.357223 18706 leader_election.cc:288] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:25.357342 18711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:25.357558 18711 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [term 1 LEADER]: Becoming Leader. State: Replica: cf8063d8897f4e999099a675c7fb91c5, State: Running, Role: LEADER
I20260927 16:28:25.357872 18711 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } }
I20260927 16:28:25.358503 18706 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:25.359071 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8063d8897f4e999099a675c7fb91c5. Latest consensus state: current_term: 1 leader_uuid: "cf8063d8897f4e999099a675c7fb91c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } } }
I20260927 16:28:25.359197 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:25.359550 18713 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8063d8897f4e999099a675c7fb91c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8063d8897f4e999099a675c7fb91c5" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40091 } } }
I20260927 16:28:25.359643 18713 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:25.361102 18718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:25.364238 18718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:25.370270 18718 catalog_manager.cc:1347] Generated new cluster ID: 630070ffeb6c425085fec8e8715d8be8
I20260927 16:28:25.370357 18718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:25.376464 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:25.371278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.15.93.190:40091)
0927 16:28:25.371395 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:25.371399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:25.371416 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:28:25.371455 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:25.371459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:25.371500 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:25.371581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:25.372171 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:25.372928 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:25.374115 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:25.374475 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:25.374562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:25.375520 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:25.375523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:25.375526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:25.376024 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:25.376056 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:25.376074 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:25.376166 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:25.376286 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 16:28:25.376339 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:28:25.388659 18718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:25.389873 18718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:25.397277 18718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8063d8897f4e999099a675c7fb91c5: Generated new TSK 0
I20260927 16:28:25.398011 18718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:28:25.407466 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18638
2026-09-27T16:28:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1398 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18739](info): set up 2 sockets
Sep 27 16:28:25 dist-test-slave-grzf krb5kdc[18739](info): commencing operation
krb5kdc: starting...
W20260927 16:28:26.447747 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:26 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:26.562520 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41161
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:26.668941 18752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:26.669283 18752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:26.669363 18752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:26.673321 18752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:26.673372 18752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:26.673393 18752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:26.673408 18752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:26.673472 18752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:26.678704 18752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:26.679711 18752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:26.680680 18752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:26.688707 18757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:26.688709 18760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:26.688756 18758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:26.689177 18752 server_base.cc:1034] running on GCE node
I20260927 16:28:26.689770 18752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:26.690774 18752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:26.691922 18752 hybrid_clock.cc:648] HybridClock initialized: now 1790526506691895 us; error 39 us; skew 500 ppm
Sep 27 16:28:26 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:26.694424 18752 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:26.695269 18752 webserver.cc:460] Webserver started at http://127.15.93.190:44011/ using document root <none> and password file <none>
I20260927 16:28:26.695816 18752 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:26.695905 18752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:26.696127 18752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:26.698099 18752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "fe26876af40d4c0fb7563bd3f6682cad"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "5a60c1c9cf3f6c862f5c6302a7597b42"
server_key_iv: "f37c4a6ec64d88c2460c1cef1c005817"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.698640 18752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "fe26876af40d4c0fb7563bd3f6682cad"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "5a60c1c9cf3f6c862f5c6302a7597b42"
server_key_iv: "f37c4a6ec64d88c2460c1cef1c005817"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.702066 18752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:26.704414 18767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:26.705516 18752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 16:28:26.705649 18752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "fe26876af40d4c0fb7563bd3f6682cad"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "5a60c1c9cf3f6c862f5c6302a7597b42"
server_key_iv: "f37c4a6ec64d88c2460c1cef1c005817"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.705757 18752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:26.721101 18752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:26.723789 18752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:26.723984 18752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:26.730772 18752 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:41161
I20260927 16:28:26.731038 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:41161 every 8 connection(s)
I20260927 16:28:26.731721 18752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:26.734284 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:26.738271 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18752
I20260927 16:28:26.738412 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:26.738699 15734 external_mini_cluster.cc:1428] Setting key 704aebe3e51546ac057649288d735168
I20260927 16:28:26.739766 18820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad: Bootstrap starting.
I20260927 16:28:26.742321 18820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:26.743125 18820 log.cc:824] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:26.745056 18820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad: No bootstrap required, opened a new log
Sep 27 16:28:26 dist-test-slave-grzf krb5kdc[18739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:26.749367 18820 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } }
I20260927 16:28:26.749619 18820 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:26.749702 18820 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe26876af40d4c0fb7563bd3f6682cad, State: Initialized, Role: FOLLOWER
I20260927 16:28:26.750168 18820 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } }
I20260927 16:28:26.750288 18820 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:26.750370 18820 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:26.750469 18820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:26.751277 18820 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } }
I20260927 16:28:26.751618 18820 leader_election.cc:302] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe26876af40d4c0fb7563bd3f6682cad; no voters:
I20260927 16:28:26.751937 18820 leader_election.cc:288] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:26.752009 18825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:26.752226 18825 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [term 1 LEADER]: Becoming Leader. State: Replica: fe26876af40d4c0fb7563bd3f6682cad, State: Running, Role: LEADER
I20260927 16:28:26.752724 18825 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } }
I20260927 16:28:26.753242 18820 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:26.753351 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:26.740031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.15.93.190:41161)
0927 16:28:26.740447 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:26.740464 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:26.740503 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:28:26.741338 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:26.741360 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:26.741756 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:26.742097 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:26.742821 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:26.744207 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:26.745039 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:26.745706 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:26.745862 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:26.748520 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:26.748524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:26.748536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:26.748561 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:26.750309 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:26.750812 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:26.750818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:26.750823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:26.750934 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:26.751215 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:26.751219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:26.751221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:26.751677 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:26.751881 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:26.752141 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 16:28:26.752429 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260927 16:28:26.754323 18827 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe26876af40d4c0fb7563bd3f6682cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } } }
I20260927 16:28:26.754442 18827 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:26.754760 18826 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe26876af40d4c0fb7563bd3f6682cad. Latest consensus state: current_term: 1 leader_uuid: "fe26876af40d4c0fb7563bd3f6682cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe26876af40d4c0fb7563bd3f6682cad" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41161 } } }
I20260927 16:28:26.754853 18826 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:26.755575 18834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:26.758072 18834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:26.763414 18834 catalog_manager.cc:1347] Generated new cluster ID: cfe0fb3f91084f21b77d4c79d0847eae
I20260927 16:28:26.763473 18834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:26.779570 18834 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:26.780454 18834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:26.787031 18834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe26876af40d4c0fb7563bd3f6682cad: Generated new TSK 0
I20260927 16:28:26.787758 18834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:26.817910 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:26.932600 18846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:26.932893 18846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:26.932999 18846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:26.937103 18846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:26.937160 18846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:26.937247 18846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:26.942420 18846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:26.943594 18846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:26.944599 18846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:26.954026 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:26.954026 18854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:26.954098 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:26.954082 18846 server_base.cc:1034] running on GCE node
I20260927 16:28:26.954728 18846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:26.955498 18846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:26.956670 18846 hybrid_clock.cc:648] HybridClock initialized: now 1790526506956660 us; error 29 us; skew 500 ppm
Sep 27 16:28:26 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:26.959345 18846 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:26.960187 18846 webserver.cc:460] Webserver started at http://127.15.93.129:46535/ using document root <none> and password file <none>
I20260927 16:28:26.960731 18846 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:26.960820 18846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:26.961030 18846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:26.962994 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "7677ad64c7b54853b582e6fb2e247bf9"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "b113113d4014be661949030f9a1bd168"
server_key_iv: "e0d3141c0e6721a9d317cce7992e2456"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.963608 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "7677ad64c7b54853b582e6fb2e247bf9"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "b113113d4014be661949030f9a1bd168"
server_key_iv: "e0d3141c0e6721a9d317cce7992e2456"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.967083 18846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:26.969493 18861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:26.970690 18846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:26.970818 18846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "7677ad64c7b54853b582e6fb2e247bf9"
format_stamp: "Formatted at 2026-09-27 16:28:26 on dist-test-slave-grzf"
server_key: "b113113d4014be661949030f9a1bd168"
server_key_iv: "e0d3141c0e6721a9d317cce7992e2456"
server_key_version: "encryptionkey@0"
I20260927 16:28:26.970927 18846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:26.983065 18846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:26.985783 18846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:26.985980 18846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:26.986531 18846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:26.987316 18846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:26.987408 18846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:26.987498 18846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:26.987552 18846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:26.996188 18846 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:34055
I20260927 16:28:26.996207 18974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:34055 every 8 connection(s)
I20260927 16:28:26.997087 18846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:27.004216 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18846
I20260927 16:28:27.004367 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:27.004665 15734 external_mini_cluster.cc:1428] Setting key 9b393b176a3e944c33632925b031fb42
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:27.011865 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:26.998888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47373 (local address 127.15.93.190:41161)
0927 16:28:26.999026 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:26.999031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:26.999906 (+ 875us) server_negotiation.cc:407] Connection header received
0927 16:28:27.001254 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:27.001258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:27.001311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:27.001398 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:27.002525 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.003054 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.004269 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.004595 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.004710 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.007803 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:27.007810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:27.007816 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:27.007849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:27.009522 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.010224 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.010227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.010228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.010299 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.010714 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.010717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.010718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.010882 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:27.010954 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:27.011648 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 16:28:27.011747 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:28:27.013113 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:26.999253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41161 (local address 127.15.93.129:47373)
0927 16:28:26.999762 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:26.999806 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:27.001050 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:27.001574 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:27.001580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:27.001930 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:27.002385 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.002397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.003185 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.003189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.004144 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.004155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.004747 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.004752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.004921 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.005574 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:27.005595 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:27.007603 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:27.009655 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.009664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.009682 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.010073 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.010393 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.010400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.010403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.010600 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.010987 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:27.010994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:27.011338 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 16:28:27.011591 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 16:28:27.011990 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
I20260927 16:28:27.014394 18975 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41161
I20260927 16:28:27.014652 18975 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:27.015169 18975 heartbeater.cc:511] Master 127.15.93.190:41161 requested a full tablet report, sending...
I20260927 16:28:27.016656 18784 ts_manager.cc:194] Registered new tserver with Master: 7677ad64c7b54853b582e6fb2e247bf9 (127.15.93.129:34055)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:27.018100 18784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:47373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:27.032207 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:27.142388 18980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:27.142643 18980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:27.142717 18980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:27.146790 18980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:27.146848 18980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:27.146944 18980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:27.152251 18980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:27.153278 18980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:27.154385 18980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:27.163884 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:27.163878 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:27.163936 18980 server_base.cc:1034] running on GCE node
W20260927 16:28:27.163913 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:27.164489 18980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:27.165061 18980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:27.166249 18980 hybrid_clock.cc:648] HybridClock initialized: now 1790526507166234 us; error 34 us; skew 500 ppm
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:27.168920 18980 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:27.169746 18980 webserver.cc:460] Webserver started at http://127.15.93.130:34579/ using document root <none> and password file <none>
I20260927 16:28:27.170362 18980 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:27.170450 18980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:27.170668 18980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:27.172667 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "6d73e01a7d4e40e69f7bc326317b7c90"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "d6591f93fbdd34e186ac56ba4a411e1c"
server_key_iv: "43916514f11d0ef0da42693c0a482cdb"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.173233 18980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "6d73e01a7d4e40e69f7bc326317b7c90"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "d6591f93fbdd34e186ac56ba4a411e1c"
server_key_iv: "43916514f11d0ef0da42693c0a482cdb"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.176956 18980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:27.179186 18995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.180477 18980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:27.180594 18980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "6d73e01a7d4e40e69f7bc326317b7c90"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "d6591f93fbdd34e186ac56ba4a411e1c"
server_key_iv: "43916514f11d0ef0da42693c0a482cdb"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.180708 18980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:27.203895 18980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:27.206763 18980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:27.206965 18980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:27.207571 18980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:27.208364 18980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:27.208431 18980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.208499 18980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:27.208535 18980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.217475 18980 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34465
I20260927 16:28:27.217499 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34465 every 8 connection(s)
I20260927 16:28:27.218329 18980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:27.221101 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 18980
I20260927 16:28:27.221220 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:27.221487 15734 external_mini_cluster.cc:1428] Setting key fc7335b9d1f71ecbac867c90606b3436
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:27.232524 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:27.220092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:36417 (local address 127.15.93.190:41161)
0927 16:28:27.220225 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:27.220229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:27.221029 (+ 800us) server_negotiation.cc:407] Connection header received
0927 16:28:27.222565 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:27.222569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:27.222620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:27.222704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:27.223870 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.224391 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.225719 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.226016 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.226133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.229042 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:27.229045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:27.229047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:27.229069 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:27.230470 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.231051 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.231053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.231054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.231155 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.231548 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.231550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.231551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.231716 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:27.231792 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:27.232282 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 16:28:27.232406 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:27.233341 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:27.220468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41161 (local address 127.15.93.130:36417)
0927 16:28:27.220887 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:27.220929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:27.222358 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:27.222864 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:27.222870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:27.223218 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:27.223733 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.223746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.224514 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.224520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.225599 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.225613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.226167 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.226171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.226355 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.226942 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:27.226957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:27.228883 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:27.230589 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.230597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.230615 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.230956 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.231259 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.231263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.231265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.231476 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.231805 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:27.231809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:27.232023 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 16:28:27.232215 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 16:28:27.232471 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:27.234576 19109 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41161
I20260927 16:28:27.234826 19109 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:27.235301 19109 heartbeater.cc:511] Master 127.15.93.190:41161 requested a full tablet report, sending...
I20260927 16:28:27.236349 18784 ts_manager.cc:194] Registered new tserver with Master: 6d73e01a7d4e40e69f7bc326317b7c90 (127.15.93.130:34465)
I20260927 16:28:27.236912 18784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:36417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:27.248371 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:27.365306 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:27.365536 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:27.365576 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:27.369553 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:27.369607 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:27.369696 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:27.375183 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:27.376214 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:27.377185 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:27.386466 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:27.386448 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:27.386440 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:27.386991 19114 server_base.cc:1034] running on GCE node
I20260927 16:28:27.387418 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:27.388025 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:27.389211 19114 hybrid_clock.cc:648] HybridClock initialized: now 1790526507389184 us; error 46 us; skew 500 ppm
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:27.391865 19114 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:27.392730 19114 webserver.cc:460] Webserver started at http://127.15.93.131:42041/ using document root <none> and password file <none>
I20260927 16:28:27.393263 19114 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:27.393322 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:27.393522 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:27.395531 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "f004fa5fe07c4847a0bbd4be8474eb9e"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "eb6ce11128f2a5a860f0e51de3406aba"
server_key_iv: "a1e205a56b811900cf0b5f84d7e0fb0b"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.396076 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "f004fa5fe07c4847a0bbd4be8474eb9e"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "eb6ce11128f2a5a860f0e51de3406aba"
server_key_iv: "a1e205a56b811900cf0b5f84d7e0fb0b"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.399780 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:27.402266 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.403568 19114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:27.403692 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "f004fa5fe07c4847a0bbd4be8474eb9e"
format_stamp: "Formatted at 2026-09-27 16:28:27 on dist-test-slave-grzf"
server_key: "eb6ce11128f2a5a860f0e51de3406aba"
server_key_iv: "a1e205a56b811900cf0b5f84d7e0fb0b"
server_key_version: "encryptionkey@0"
I20260927 16:28:27.403797 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:27.421114 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:27.424276 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:27.424469 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:27.425097 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:27.425882 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:27.425949 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.426019 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:27.426071 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:27.436363 19114 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:44087
I20260927 16:28:27.436432 19242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:44087 every 8 connection(s)
I20260927 16:28:27.437220 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:27.445031 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19114
I20260927 16:28:27.445160 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:27.445477 15734 external_mini_cluster.cc:1428] Setting key c146cb3b02d88f824adacf37c96a4090
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:27.452140 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:27.438937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:52249 (local address 127.15.93.190:41161)
0927 16:28:27.439088 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:27.439094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:27.439962 (+ 868us) server_negotiation.cc:407] Connection header received
0927 16:28:27.441292 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:27.441298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:27.441367 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:27.441473 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:27.442560 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.443570 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.444654 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.445114 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.445243 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.448282 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:27.448288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:27.448294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:27.448328 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:27.449998 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.450470 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.450476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.450481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.450577 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.450877 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.450882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.450886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.451151 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:27.451267 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:27.451750 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 16:28:27.451891 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:28:27.452741 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:27.439304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41161 (local address 127.15.93.131:52249)
0927 16:28:27.439804 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:27.439853 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:27.441081 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:27.441628 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:27.441652 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:27.441977 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:27.442423 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.442435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.443690 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.443692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.444536 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:27.444545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.445347 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.445353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:27.445542 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.446285 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:27.446306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:27.448102 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:27.450110 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.450114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.450125 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.450365 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.450668 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:27.450670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:27.450671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:27.450790 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:27.451256 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:27.451260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:27.451526 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 16:28:27.451676 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 16:28:27.451898 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260927 16:28:27.453804 19243 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41161
I20260927 16:28:27.454038 19243 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:27.454591 19243 heartbeater.cc:511] Master 127.15.93.190:41161 requested a full tablet report, sending...
I20260927 16:28:27.455627 18784 ts_manager.cc:194] Registered new tserver with Master: f004fa5fe07c4847a0bbd4be8474eb9e (127.15.93.131:44087)
I20260927 16:28:27.456182 18784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:52249
I20260927 16:28:27.459333 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[18739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526507, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:27.485656 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:27.477576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.15.93.190:41161)
0927 16:28:27.477736 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:27.477741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:27.477878 (+ 137us) server_negotiation.cc:407] Connection header received
0927 16:28:27.478051 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:27.478054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:27.478095 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:27.478185 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:27.478871 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.479450 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.480274 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:27.480493 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:27.480603 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:27.482849 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:27.482859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:27.482862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:27.482883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:27.484162 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.484592 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.484595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.484596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.484679 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:27.484909 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:27.484911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:27.484912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:27.485106 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:27.485223 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:27.485430 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 16:28:27.485529 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 16:28:27.486094 18784 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41468
I20260927 16:28:27.486989 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18846
I20260927 16:28:27.493358 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18980
I20260927 16:28:27.498876 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19114
I20260927 16:28:27.504046 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 18752
2026-09-27T16:28:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2098 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[19255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[19255](info): set up 2 sockets
Sep 27 16:28:27 dist-test-slave-grzf krb5kdc[19255](info): commencing operation
krb5kdc: starting...
W20260927 16:28:28.543761 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:28 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:28.655987 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38249
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:28.761169 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:28.761415 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:28.761454 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:28.765461 19268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:28.765522 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:28.765542 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:28.765558 19268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:28.765573 19268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:28.770661 19268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:28.771622 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:28.772542 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:28.781296 19273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:28.781316 19274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:28.781375 19268 server_base.cc:1034] running on GCE node
W20260927 16:28:28.781296 19276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:28.782047 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:28.783035 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:28.784183 19268 hybrid_clock.cc:648] HybridClock initialized: now 1790526508784154 us; error 41 us; skew 500 ppm
Sep 27 16:28:28 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526508, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:28.786659 19268 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:28.787565 19268 webserver.cc:460] Webserver started at http://127.15.93.190:40487/ using document root <none> and password file <none>
I20260927 16:28:28.788093 19268 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:28.788183 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:28.788407 19268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:28.790303 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "8b2c11ba7bbd4278bcb59983fc65d579"
format_stamp: "Formatted at 2026-09-27 16:28:28 on dist-test-slave-grzf"
server_key: "965bb29a2510abed63b09c75dbe395fe"
server_key_iv: "01371b574046783a9a26ae5d108573f4"
server_key_version: "encryptionkey@0"
I20260927 16:28:28.790850 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "8b2c11ba7bbd4278bcb59983fc65d579"
format_stamp: "Formatted at 2026-09-27 16:28:28 on dist-test-slave-grzf"
server_key: "965bb29a2510abed63b09c75dbe395fe"
server_key_iv: "01371b574046783a9a26ae5d108573f4"
server_key_version: "encryptionkey@0"
I20260927 16:28:28.794324 19268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:28.796594 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:28.797744 19268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:28.797875 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "8b2c11ba7bbd4278bcb59983fc65d579"
format_stamp: "Formatted at 2026-09-27 16:28:28 on dist-test-slave-grzf"
server_key: "965bb29a2510abed63b09c75dbe395fe"
server_key_iv: "01371b574046783a9a26ae5d108573f4"
server_key_version: "encryptionkey@0"
I20260927 16:28:28.797978 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:28.808100 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:28.810632 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:28.810822 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:28.817813 19268 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38249
I20260927 16:28:28.817834 19335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38249 every 8 connection(s)
I20260927 16:28:28.818693 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:28.821148 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19268
I20260927 16:28:28.821280 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:28.821429 19336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:28.821588 15734 external_mini_cluster.cc:1428] Setting key bc7198b00f3a81c7499ab65ff1c9bfd4
I20260927 16:28:28.827641 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579: Bootstrap starting.
I20260927 16:28:28.829943 19336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579: Neither blocks nor log segments found. Creating new log.
Sep 27 16:28:28 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:28.830835 19336 log.cc:824] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:28.832934 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579: No bootstrap required, opened a new log
I20260927 16:28:28.835361 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:28.822957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.15.93.190:38249)
0927 16:28:28.823364 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:28.823420 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:28.823458 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:28:28.824147 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:28.824164 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:28.824475 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:28.824869 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:28.825679 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:28.826679 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:28.827600 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:28.828130 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:28.828299 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:28.830974 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:28.830984 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:28.830996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:28.831022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:28.832701 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:28.833176 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:28.833181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:28.833185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:28.833291 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:28.833572 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:28.833574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:28.833575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:28.833935 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:28.834080 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:28.834365 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 16:28:28.834680 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
I20260927 16:28:28.837363 19336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } }
I20260927 16:28:28.837604 19336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:28.837704 19336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2c11ba7bbd4278bcb59983fc65d579, State: Initialized, Role: FOLLOWER
I20260927 16:28:28.838136 19336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } }
I20260927 16:28:28.838286 19336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:28.838347 19336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:28.838447 19336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:28.839313 19336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } }
I20260927 16:28:28.839720 19336 leader_election.cc:302] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b2c11ba7bbd4278bcb59983fc65d579; no voters:
I20260927 16:28:28.840018 19336 leader_election.cc:288] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:28.840142 19341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:28.840396 19341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [term 1 LEADER]: Becoming Leader. State: Replica: 8b2c11ba7bbd4278bcb59983fc65d579, State: Running, Role: LEADER
I20260927 16:28:28.840735 19341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } }
I20260927 16:28:28.841210 19336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:28.842330 19343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b2c11ba7bbd4278bcb59983fc65d579. Latest consensus state: current_term: 1 leader_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } } }
I20260927 16:28:28.842464 19343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:28.842846 19350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:28.843478 19342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2c11ba7bbd4278bcb59983fc65d579" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38249 } } }
I20260927 16:28:28.843585 19342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:28.845983 19350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:28.851562 19350 catalog_manager.cc:1347] Generated new cluster ID: b95888ee5eba45e1a37307bc5b4046a7
I20260927 16:28:28.851672 19350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:28.869453 19350 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:28.870641 19350 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:28.878472 19350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b2c11ba7bbd4278bcb59983fc65d579: Generated new TSK 0
I20260927 16:28:28.879158 19350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:28.907891 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:29.028867 19362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:29.029127 19362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:29.029204 19362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:29.033072 19362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:29.033154 19362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:29.033241 19362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:29.038303 19362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:29.039320 19362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:29.040321 19362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:29.049623 19370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:29.049620 19368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:29.049616 19367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:29.049916 19362 server_base.cc:1034] running on GCE node
I20260927 16:28:29.050340 19362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:29.050889 19362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:29.052040 19362 hybrid_clock.cc:648] HybridClock initialized: now 1790526509052027 us; error 24 us; skew 500 ppm
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:29.054605 19362 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:29.055510 19362 webserver.cc:460] Webserver started at http://127.15.93.129:33093/ using document root <none> and password file <none>
I20260927 16:28:29.056169 19362 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:29.056254 19362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:29.056455 19362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:29.058349 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "1c54494594de42f696905eb95a7ba701"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "74a430f9dc8e58d8b769bd87e4a4eb3b"
server_key_iv: "f615b7e9f1a24d17af3bd4623e5953b1"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.058902 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "1c54494594de42f696905eb95a7ba701"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "74a430f9dc8e58d8b769bd87e4a4eb3b"
server_key_iv: "f615b7e9f1a24d17af3bd4623e5953b1"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.062430 19362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:28:29.064800 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.065953 19362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:29.066082 19362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "1c54494594de42f696905eb95a7ba701"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "74a430f9dc8e58d8b769bd87e4a4eb3b"
server_key_iv: "f615b7e9f1a24d17af3bd4623e5953b1"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.066187 19362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:29.080397 19362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:29.083016 19362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:29.083210 19362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:29.083865 19362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:29.084645 19362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:29.084713 19362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.084774 19362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:29.084822 19362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.093024 19362 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:35309
I20260927 16:28:29.093051 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:35309 every 8 connection(s)
I20260927 16:28:29.093878 19362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:29.094394 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19362
I20260927 16:28:29.094556 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:29.094856 15734 external_mini_cluster.cc:1428] Setting key 5e8e1ad3f6a472f29d4397adce8ec111
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:29.108181 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.095880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:41609 (local address 127.15.93.190:38249)
0927 16:28:29.096028 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:29.096032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:29.097038 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 16:28:29.098374 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:29.098377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:29.098425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:29.098506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:29.099716 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.100232 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.101468 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.101710 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.101869 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.104530 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:29.104533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:29.104535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:29.104556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:29.105920 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.106581 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.106585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.106586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.106698 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.107029 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.107035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.107040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.107286 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:29.107422 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:29.107885 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 16:28:29.108007 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:28:29.109030 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.096423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38249 (local address 127.15.93.129:41609)
0927 16:28:29.096898 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:29.096946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:29.098179 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:29.098685 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:29.098704 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:29.099022 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:29.099554 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.099568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.100353 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.100361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.101351 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.101360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.101905 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.101909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.102072 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.102701 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:29.102717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:29.104376 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:29.106049 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.106056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.106088 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.106468 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.106795 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.106798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.106799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.106932 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.107430 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:29.107436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:29.107655 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 16:28:29.107837 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 16:28:29.108149 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260927 16:28:29.110251 19491 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38249
I20260927 16:28:29.110476 19491 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:29.110915 19491 heartbeater.cc:511] Master 127.15.93.190:38249 requested a full tablet report, sending...
I20260927 16:28:29.112353 19300 ts_manager.cc:194] Registered new tserver with Master: 1c54494594de42f696905eb95a7ba701 (127.15.93.129:35309)
I20260927 16:28:29.113433 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:41609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:29.120362 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:29.226809 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:29.227044 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:29.227103 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:29.231022 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:29.231077 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:29.231132 19496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:29.236217 19496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:29.237156 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:29.238086 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:29.248054 19502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:29.248054 19501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:29.248085 19504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:29.248373 19496 server_base.cc:1034] running on GCE node
I20260927 16:28:29.248826 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:29.249442 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:29.250640 19496 hybrid_clock.cc:648] HybridClock initialized: now 1790526509250606 us; error 45 us; skew 500 ppm
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:29.253466 19496 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:29.254362 19496 webserver.cc:460] Webserver started at http://127.15.93.130:34093/ using document root <none> and password file <none>
I20260927 16:28:29.254938 19496 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:29.255036 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:29.255293 19496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:29.257334 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "13b1af2d171746e8a267d52f42fb1674"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "057ab79d526f929567483d3f6493ac26"
server_key_iv: "5f95196c9f04dc1fa3be7e885537e19f"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.257917 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "13b1af2d171746e8a267d52f42fb1674"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "057ab79d526f929567483d3f6493ac26"
server_key_iv: "5f95196c9f04dc1fa3be7e885537e19f"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.261579 19496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:29.264058 19511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.265254 19496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:29.265388 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "13b1af2d171746e8a267d52f42fb1674"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "057ab79d526f929567483d3f6493ac26"
server_key_iv: "5f95196c9f04dc1fa3be7e885537e19f"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.265501 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:29.282114 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:29.284787 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:29.284978 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:29.285524 19496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:29.286321 19496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:29.286391 19496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.286466 19496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:29.286510 19496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.294955 19496 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:42295
I20260927 16:28:29.294981 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:42295 every 8 connection(s)
I20260927 16:28:29.295832 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:29.306056 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19496
I20260927 16:28:29.306203 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:29.306536 15734 external_mini_cluster.cc:1428] Setting key 2f509db77845b8bf4d6217154eb9860c
I20260927 16:28:29.309134 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.297555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41771 (local address 127.15.93.190:38249)
0927 16:28:29.297724 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:29.297729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:29.298469 (+ 740us) server_negotiation.cc:407] Connection header received
0927 16:28:29.299802 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:29.299805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:29.299852 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:29.299924 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:29.301049 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.301609 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.302833 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.303044 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.303123 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.305485 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:29.305488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:29.305490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:29.305512 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:29.307048 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.307593 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.307597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.307599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.307690 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.308063 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.308067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.308068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.308256 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:29.308361 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:29.308830 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 16:28:29.308956 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 16:28:29.309921 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.297911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38249 (local address 127.15.93.130:41771)
0927 16:28:29.298320 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:29.298359 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:29.299612 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:29.300074 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:29.300080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:29.300412 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:29.300928 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.300942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.301728 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.301730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.302732 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.302742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.303178 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.303181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.303319 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.303952 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:29.303967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:29.305341 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:29.307176 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.307180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.307191 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.307490 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.307793 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.307796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.307798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.307958 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.308370 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:29.308379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:29.308593 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 16:28:29.308774 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:28:29.309065 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260927 16:28:29.311133 19625 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38249
I20260927 16:28:29.311357 19625 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:29.311823 19625 heartbeater.cc:511] Master 127.15.93.190:38249 requested a full tablet report, sending...
I20260927 16:28:29.312821 19300 ts_manager.cc:194] Registered new tserver with Master: 13b1af2d171746e8a267d52f42fb1674 (127.15.93.130:42295)
I20260927 16:28:29.313395 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:41771
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:29.331542 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:29.438483 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:29.438743 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:29.438786 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:29.442871 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:29.442922 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:29.442977 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:29.448266 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:29.449144 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:29.450027 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:29.459039 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:29.459038 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:29.459084 19630 server_base.cc:1034] running on GCE node
W20260927 16:28:29.459038 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:29.459649 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:29.460260 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:29.461433 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790526509461409 us; error 47 us; skew 500 ppm
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:29.464181 19630 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:29.464967 19630 webserver.cc:460] Webserver started at http://127.15.93.131:40419/ using document root <none> and password file <none>
I20260927 16:28:29.465484 19630 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:29.465574 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:29.465797 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:29.467767 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "dc35ca2bbd1741d0946a158714f8a061"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "b5fa0f2ff488cc87bd001641f47aafc5"
server_key_iv: "22c4001a448b87d2bc67e3ed3edb6a60"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.468354 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "dc35ca2bbd1741d0946a158714f8a061"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "b5fa0f2ff488cc87bd001641f47aafc5"
server_key_iv: "22c4001a448b87d2bc67e3ed3edb6a60"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.472127 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 16:28:29.474381 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.475623 19630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:29.475751 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "dc35ca2bbd1741d0946a158714f8a061"
format_stamp: "Formatted at 2026-09-27 16:28:29 on dist-test-slave-grzf"
server_key: "b5fa0f2ff488cc87bd001641f47aafc5"
server_key_iv: "22c4001a448b87d2bc67e3ed3edb6a60"
server_key_version: "encryptionkey@0"
I20260927 16:28:29.475853 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:29.498935 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:29.502168 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:29.502358 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:29.502871 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:29.503674 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:29.503719 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.503757 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:29.503778 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:29.511833 19630 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:45721
I20260927 16:28:29.511894 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:45721 every 8 connection(s)
I20260927 16:28:29.512705 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:29.517356 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19630
I20260927 16:28:29.517441 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:29.517719 15734 external_mini_cluster.cc:1428] Setting key 9fd02505dea2e6ad972a3c6bde5085ef
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:29.525492 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.514376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:45115 (local address 127.15.93.190:38249)
0927 16:28:29.514520 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:29.514524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:29.515290 (+ 766us) server_negotiation.cc:407] Connection header received
0927 16:28:29.516694 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:29.516697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:29.516753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:29.516863 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:29.518078 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.518604 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.519822 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.520044 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.520160 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.522364 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:29.522368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:29.522370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:29.522390 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:29.523605 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.524071 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.524076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.524078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.524161 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.524477 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.524482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.524484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.524668 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:29.524772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:29.525223 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 16:28:29.525336 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:28:29.526188 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.514742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38249 (local address 127.15.93.131:45115)
0927 16:28:29.515153 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:29.515192 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:29.516463 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:29.517028 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:29.517034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:29.517375 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:29.517946 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.517960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.518722 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.518728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.519706 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:29.519715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.520185 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.520189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:29.520348 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.520822 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:29.520838 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:29.522210 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:29.523714 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.523719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.523732 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.523972 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.524251 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:29.524255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:29.524257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:29.524384 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:29.524769 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:29.524775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:29.524995 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 16:28:29.525169 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 16:28:29.525452 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260927 16:28:29.527141 19759 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38249
I20260927 16:28:29.527359 19759 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:29.527820 19759 heartbeater.cc:511] Master 127.15.93.190:38249 requested a full tablet report, sending...
I20260927 16:28:29.528764 19300 ts_manager.cc:194] Registered new tserver with Master: dc35ca2bbd1741d0946a158714f8a061 (127.15.93.131:45721)
I20260927 16:28:29.529268 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:45115
I20260927 16:28:29.531147 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:29.555806 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.546966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.15.93.129:35309)
0927 16:28:29.547294 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:29.547303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:29.547327 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:28:29.547414 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:29.547420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:29.547619 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:29.547752 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:29.548378 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.549140 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.549913 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.550264 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.550344 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.552545 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:29.552553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:29.552566 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:29.552590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:29.554194 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.554566 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.554571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.554572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.554657 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.554877 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.554879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.554880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.555094 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:29.555240 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:29.555445 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 16:28:29.555601 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:29.566212 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:29.558436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.15.93.190:38249)
0927 16:28:29.558575 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:29.558581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:29.558596 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:29.558723 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:29.558726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:29.558775 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:29.558860 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:29.559448 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.559956 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.560719 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:29.560939 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:29.561046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:29.563244 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:29.563252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:29.563254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:29.563277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:29.564616 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.565019 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.565024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.565026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.565103 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:29.565376 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:29.565379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:29.565381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:29.565582 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:29.565727 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:29.565940 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 16:28:29.566053 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 16:28:29.566689 19300 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60000
I20260927 16:28:29.567265 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19362
I20260927 16:28:29.573258 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19496
I20260927 16:28:29.578400 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19630
I20260927 16:28:29.583751 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19268
2026-09-27T16:28:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2080 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19771](info): set up 2 sockets
Sep 27 16:28:29 dist-test-slave-grzf krb5kdc[19771](info): commencing operation
krb5kdc: starting...
W20260927 16:28:30.616046 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:30 dist-test-slave-grzf krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:30.732167 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:32999
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:30.840456 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:30.840703 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:30.840780 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:30.844789 19784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:30.844841 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:30.844861 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:30.844909 19784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:30.844940 19784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:30.850327 19784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:30.851640 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:30.852670 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:30.860495 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:30.860535 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:30.860704 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:30.860558 19784 server_base.cc:1034] running on GCE node
I20260927 16:28:30.861266 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:30.862278 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:30.863458 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790526510863431 us; error 31 us; skew 500 ppm
Sep 27 16:28:30 dist-test-slave-grzf krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526510, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:30.872495 19784 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:30.873191 19784 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:30.873253 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:30.873425 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:30.875414 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "d4b37750bb3e437fa211382fa7bf8c47"
format_stamp: "Formatted at 2026-09-27 16:28:30 on dist-test-slave-grzf"
server_key: "fa6972ba847f9d60a5bc53c768e25517"
server_key_iv: "34db2ad882a0cb92496299ac5e5bb45d"
server_key_version: "encryptionkey@0"
I20260927 16:28:30.875945 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "d4b37750bb3e437fa211382fa7bf8c47"
format_stamp: "Formatted at 2026-09-27 16:28:30 on dist-test-slave-grzf"
server_key: "fa6972ba847f9d60a5bc53c768e25517"
server_key_iv: "34db2ad882a0cb92496299ac5e5bb45d"
server_key_version: "encryptionkey@0"
I20260927 16:28:30.879554 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:30.881769 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:30.882947 19784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:30.883083 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "d4b37750bb3e437fa211382fa7bf8c47"
format_stamp: "Formatted at 2026-09-27 16:28:30 on dist-test-slave-grzf"
server_key: "fa6972ba847f9d60a5bc53c768e25517"
server_key_iv: "34db2ad882a0cb92496299ac5e5bb45d"
server_key_version: "encryptionkey@0"
I20260927 16:28:30.883200 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:30.908129 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:30.910974 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:30.911120 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:30.917704 19784 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:32999
I20260927 16:28:30.917966 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:32999 every 8 connection(s)
I20260927 16:28:30.918375 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:30.921005 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:30.926235 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: Bootstrap starting.
I20260927 16:28:30.927908 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19784
I20260927 16:28:30.928054 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:30.928325 15734 external_mini_cluster.cc:1428] Setting key d0435890ae55b74a8f9679ed42c87f3d
I20260927 16:28:30.928856 19851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:30.929735 19851 log.cc:824] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:30.931524 19851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: No bootstrap required, opened a new log
I20260927 16:28:30.934796 19851 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } }
I20260927 16:28:30.935086 19851 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:30.935168 19851 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b37750bb3e437fa211382fa7bf8c47, State: Initialized, Role: FOLLOWER
I20260927 16:28:30.935614 19851 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } }
I20260927 16:28:30.935758 19851 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:30.935827 19851 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:30.935918 19851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:28:30 dist-test-slave-grzf krb5kdc[19771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:30.937013 19851 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } }
I20260927 16:28:30.937341 19851 leader_election.cc:302] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b37750bb3e437fa211382fa7bf8c47; no voters:
I20260927 16:28:30.937619 19851 leader_election.cc:288] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:30.937753 19856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:30.938344 19856 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [term 1 LEADER]: Becoming Leader. State: Replica: d4b37750bb3e437fa211382fa7bf8c47, State: Running, Role: LEADER
I20260927 16:28:30.938643 19856 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } }
I20260927 16:28:30.938829 19851 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:30.940119 19858 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b37750bb3e437fa211382fa7bf8c47. Latest consensus state: current_term: 1 leader_uuid: "d4b37750bb3e437fa211382fa7bf8c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } } }
I20260927 16:28:30.940244 19858 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:30.940589 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b37750bb3e437fa211382fa7bf8c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b37750bb3e437fa211382fa7bf8c47" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 32999 } } }
I20260927 16:28:30.940691 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:30.943185 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:30.929395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.15.93.190:32999)
0927 16:28:30.929810 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:30.929824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:30.929881 (+ 57us) server_negotiation.cc:407] Connection header received
0927 16:28:30.930488 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:30.930503 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:30.930805 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:30.931126 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:30.931945 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:30.932876 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:30.933725 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:30.934187 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:30.934356 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:30.937231 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:30.937236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:30.937248 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:30.937272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:30.939465 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:30.939999 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:30.940005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:30.940011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:30.940148 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:30.940604 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:30.940607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:30.940609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:30.941092 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:30.941273 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:30.941961 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 16:28:30.942259 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
W20260927 16:28:30.944113 19871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:30.944208 19871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:30.944272 19865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:30.944931 19865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:30.949667 19865 catalog_manager.cc:1347] Generated new cluster ID: 143fc3a5250f4eb98744d420165d0e83
I20260927 16:28:30.949733 19865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:30.977916 19865 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:30.979270 19865 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:30.993254 19865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: Generated new TSK 0
I20260927 16:28:30.993937 19865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:31.027936 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:31.136430 19877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:31.136744 19877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:31.136818 19877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:31.141251 19877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:31.141333 19877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:31.141424 19877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:31.146611 19877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:31.147764 19877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:31.148770 19877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:31.158186 19885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.158192 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.158206 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:31.158228 19877 server_base.cc:1034] running on GCE node
I20260927 16:28:31.158874 19877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:31.159600 19877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:31.160785 19877 hybrid_clock.cc:648] HybridClock initialized: now 1790526511160771 us; error 33 us; skew 500 ppm
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:31.163887 19877 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:31.164639 19877 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:31.164731 19877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:31.164959 19877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:31.166923 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "4e5766fb90d84638839f779a1e6c9849"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "8f7848f9796c81caced534c2ece15495"
server_key_iv: "e2c5e1eb65923672796d999490865a56"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.167565 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "4e5766fb90d84638839f779a1e6c9849"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "8f7848f9796c81caced534c2ece15495"
server_key_iv: "e2c5e1eb65923672796d999490865a56"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.171041 19877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:31.173396 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.174576 19877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:28:31.174710 19877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "4e5766fb90d84638839f779a1e6c9849"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "8f7848f9796c81caced534c2ece15495"
server_key_iv: "e2c5e1eb65923672796d999490865a56"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.174815 19877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:31.196473 19877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:31.199180 19877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:31.199402 19877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:31.199960 19877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:31.200770 19877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:31.200840 19877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.200912 19877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:31.200965 19877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.209643 19877 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:41273
I20260927 16:28:31.209672 20004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:41273 every 8 connection(s)
I20260927 16:28:31.210210 19877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:31.213717 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 19877
I20260927 16:28:31.213862 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:31.214140 15734 external_mini_cluster.cc:1428] Setting key a55262d35346abe0e4ff1ee8c6cb7ebf
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:31.225006 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.211975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:42095 (local address 127.15.93.190:32999)
0927 16:28:31.212113 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.212117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.212925 (+ 808us) server_negotiation.cc:407] Connection header received
0927 16:28:31.214244 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.214248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.214302 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.214400 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:31.215743 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.216533 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.218097 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.218414 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.218529 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.221268 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:31.221274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:31.221279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:31.221313 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:31.222990 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.223519 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.223525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.223529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.223625 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.223962 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.223965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.223966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.224190 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:31.224301 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.224782 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.224887 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:28:31.225773 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:32999 (local address 127.15.93.129:42095)
0927 16:28:31.212778 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.212819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.214039 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.214587 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.214596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.214992 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:31.215589 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.215606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.216665 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.216667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.217957 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.217970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.218554 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.218556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.218751 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.219440 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:31.219455 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:31.221088 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:31.223103 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.223107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.223125 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.223363 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.223735 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.223737 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.223739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.223861 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.224324 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:31.224328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:31.224556 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.224701 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.224939 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:31.227104 20005 heartbeater.cc:347] Connected to a master server at 127.15.93.190:32999
I20260927 16:28:31.227336 20005 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:31.227797 20005 heartbeater.cc:511] Master 127.15.93.190:32999 requested a full tablet report, sending...
I20260927 16:28:31.229305 19813 ts_manager.cc:194] Registered new tserver with Master: 4e5766fb90d84638839f779a1e6c9849 (127.15.93.129:41273)
I20260927 16:28:31.230967 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:42095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:31.240578 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:31.346446 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:31.346702 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:31.346746 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:31.350596 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:31.350643 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:31.350699 20010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:31.355703 20010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:31.356588 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:31.357472 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:31.366218 20015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.366261 20016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.366250 20018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:31.366520 20010 server_base.cc:1034] running on GCE node
I20260927 16:28:31.366962 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:31.367640 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:31.368804 20010 hybrid_clock.cc:648] HybridClock initialized: now 1790526511368782 us; error 32 us; skew 500 ppm
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:31.371520 20010 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:31.372215 20010 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:31.372306 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:31.372504 20010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:31.374433 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "4708f0c669234e3f806208a1d24ee4c0"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "d469320a5a7d743f1f6cb9eecd47156c"
server_key_iv: "7a63cfb0b6d77cd3ce3909d812103cfd"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.374996 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "4708f0c669234e3f806208a1d24ee4c0"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "d469320a5a7d743f1f6cb9eecd47156c"
server_key_iv: "7a63cfb0b6d77cd3ce3909d812103cfd"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.378537 20010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:31.380962 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.382146 20010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:31.382279 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "4708f0c669234e3f806208a1d24ee4c0"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "d469320a5a7d743f1f6cb9eecd47156c"
server_key_iv: "7a63cfb0b6d77cd3ce3909d812103cfd"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.382382 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:31.400835 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:31.404024 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:31.404214 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:31.404760 20010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:31.405560 20010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:31.405634 20010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.405722 20010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:31.405771 20010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.414175 20010 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:33973
I20260927 16:28:31.414204 20137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:33973 every 8 connection(s)
I20260927 16:28:31.414731 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:31.416147 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20010
I20260927 16:28:31.416294 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:31.416610 15734 external_mini_cluster.cc:1428] Setting key fe43182070575e15354693c4e76d3f46
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:31.428591 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.416365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37201 (local address 127.15.93.190:32999)
0927 16:28:31.416498 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.416504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.417358 (+ 854us) server_negotiation.cc:407] Connection header received
0927 16:28:31.418656 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.418660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.418714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.418817 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:31.420311 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.421007 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.422109 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.422489 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.422611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.424958 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:31.424963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:31.424966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:31.424999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:31.426491 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.427008 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.427011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.427012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.427083 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.427368 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.427422 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.427423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.427584 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:31.427667 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.428199 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.428343 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:31.429548 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.416777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:32999 (local address 127.15.93.130:37201)
0927 16:28:31.417228 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.417278 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.418445 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.418997 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.419004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.419461 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:31.420152 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.420174 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.421149 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.421152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.421993 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.422002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.422637 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.422641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.422780 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.423355 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:31.423409 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:31.424797 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:31.426589 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.426593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.426604 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.426904 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.427161 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.427164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.427165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.427289 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.427689 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:31.427722 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:31.427979 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.428163 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.428508 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":122,"threads_started":1}
I20260927 16:28:31.430753 20138 heartbeater.cc:347] Connected to a master server at 127.15.93.190:32999
I20260927 16:28:31.431000 20138 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:31.431490 20138 heartbeater.cc:511] Master 127.15.93.190:32999 requested a full tablet report, sending...
I20260927 16:28:31.432492 19813 ts_manager.cc:194] Registered new tserver with Master: 4708f0c669234e3f806208a1d24ee4c0 (127.15.93.130:33973)
I20260927 16:28:31.433039 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:37201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:31.442255 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:31.548875 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:31.549132 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:31.549177 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:31.553071 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:31.553122 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:31.553180 20143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:31.558487 20143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:32999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:31.559438 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:31.560360 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:31.569598 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.569604 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:31.569648 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:31.569834 20143 server_base.cc:1034] running on GCE node
I20260927 16:28:31.570271 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:31.570855 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:31.572009 20143 hybrid_clock.cc:648] HybridClock initialized: now 1790526511571969 us; error 46 us; skew 500 ppm
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:31.574582 20143 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:31.575263 20143 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:31.575354 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:31.575579 20143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:31.577498 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "1cbf9cb6a3564d4ba43b81fc4972a494"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "6025e995d6866e91cbeb438d975976f5"
server_key_iv: "53d500098827809758c1d50e8717e06c"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.578107 20143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "1cbf9cb6a3564d4ba43b81fc4972a494"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "6025e995d6866e91cbeb438d975976f5"
server_key_iv: "53d500098827809758c1d50e8717e06c"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.581794 20143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:28:31.584057 20157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.585253 20143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:31.585381 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "1cbf9cb6a3564d4ba43b81fc4972a494"
format_stamp: "Formatted at 2026-09-27 16:28:31 on dist-test-slave-grzf"
server_key: "6025e995d6866e91cbeb438d975976f5"
server_key_iv: "53d500098827809758c1d50e8717e06c"
server_key_version: "encryptionkey@0"
I20260927 16:28:31.585482 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:31.596786 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:31.599447 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:31.599653 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:31.600185 20143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:31.600951 20143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:31.601019 20143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.601083 20143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:31.601137 20143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:31.609587 20143 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:44381
I20260927 16:28:31.609603 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:44381 every 8 connection(s)
I20260927 16:28:31.610139 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:31.617871 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20143
I20260927 16:28:31.618021 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:31.618321 15734 external_mini_cluster.cc:1428] Setting key 4a0fc3bffcac44bbe1c169a7bd735cdf
Sep 27 16:28:31 dist-test-slave-grzf krb5kdc[19771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:31.623346 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.611760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:50363 (local address 127.15.93.190:32999)
0927 16:28:31.611893 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.611896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.612607 (+ 711us) server_negotiation.cc:407] Connection header received
0927 16:28:31.613863 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.613866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.613925 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.614032 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:31.615150 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.615758 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.616855 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.617079 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.617194 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.619880 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:31.619887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:31.619893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:31.619925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:31.621399 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.621908 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.621917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.621919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.621995 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.622318 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.622323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.622325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.622495 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:31.622600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.623063 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.623186 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:31.624092 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.612064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:32999 (local address 127.15.93.131:50363)
0927 16:28:31.612468 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.612507 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.613661 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.614200 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.614207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.614553 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:31.615005 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.615017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.615877 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.615881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.616732 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.616740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.617227 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.617231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.617394 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.617976 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:31.618015 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:31.619680 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:31.621526 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.621531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.621544 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.621798 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.622084 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:31.622089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:31.622091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:31.622225 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:31.622597 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:31.622603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:31.622838 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.623022 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.623317 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260927 16:28:31.625051 20271 heartbeater.cc:347] Connected to a master server at 127.15.93.190:32999
I20260927 16:28:31.625277 20271 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:31.625700 20271 heartbeater.cc:511] Master 127.15.93.190:32999 requested a full tablet report, sending...
I20260927 16:28:31.626634 19813 ts_manager.cc:194] Registered new tserver with Master: 1cbf9cb6a3564d4ba43b81fc4972a494 (127.15.93.131:44381)
I20260927 16:28:31.627183 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:50363
I20260927 16:28:31.631876 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:28:31.640224 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.633640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.15.93.190:32999)
0927 16:28:31.633770 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.633774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.633891 (+ 117us) server_negotiation.cc:407] Connection header received
0927 16:28:31.633980 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.633982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.634017 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.634074 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:31.634708 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.635177 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.635972 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.636173 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.636246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.637393 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:31.637396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:31.637398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:31.637418 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:31.638587 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.639015 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.639018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.639019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.639087 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:31.639286 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:31.639288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:31.639289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:31.639528 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:31.639704 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.639873 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.640043 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:28:31.642874 19813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:31.645759 19813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:31.659500 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.654304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.15.93.130:33973)
0927 16:28:31.654807 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.654821 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.654846 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:28:31.654920 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.654930 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.655143 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.655293 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:31.655732 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.656915 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.658282 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.658939 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.659060 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.659205 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.659257 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.659347 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":106,"threads_started":1}
I20260927 16:28:31.659492 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.654155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:33973 (local address 127.0.0.1:52248)
0927 16:28:31.654699 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.654711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.654786 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.655493 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.655496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.655517 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:31.655603 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.655609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.657053 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.657057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.658154 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.658164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.659059 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.659063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.659143 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.659164 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.659228 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.659295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260927 16:28:31.660152 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.653869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41273 (local address 127.0.0.1:58912)
0927 16:28:31.654446 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.654476 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.654556 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.655793 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.655796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.655814 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:31.655897 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.655902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.657179 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.657185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.658502 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.658512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.659769 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.659773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.659870 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.659907 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.659983 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.660045 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":28,"thread_start_us":112,"threads_started":1}
I20260927 16:28:31.660174 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.653575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.15.93.131:44381)
0927 16:28:31.653938 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.653950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.654532 (+ 582us) server_negotiation.cc:407] Connection header received
0927 16:28:31.654614 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.654637 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.654853 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.655010 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:31.655301 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.656332 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.657766 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.658534 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.658655 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.658796 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.659398 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.659500 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":177,"threads_started":1}
I20260927 16:28:31.660053 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.653450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44381 (local address 127.0.0.1:41242)
0927 16:28:31.654202 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.654249 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.654377 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.655006 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.655012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.655033 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:31.655159 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.655166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.656460 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.656464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.657633 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.657643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.658859 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.658862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.658978 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.659013 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.659115 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.659241 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":33,"thread_start_us":76,"threads_started":1}
I20260927 16:28:31.660413 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.654298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.15.93.129:41273)
0927 16:28:31.655414 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.655425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.655446 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:28:31.655514 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.655517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.655645 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.655803 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:31.655984 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.657041 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.658638 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.659461 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.659569 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.659706 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.660031 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.660134 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":92,"threads_started":1}
I20260927 16:28:31.662820 20072 tablet_service.cc:1467] Processing CreateTablet for tablet 85156d7df0344620b71cdb0f128720b6 (DEFAULT_TABLE table=test-table [id=c626264042aa4840b3e8de3196a748b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:31.662822 19939 tablet_service.cc:1467] Processing CreateTablet for tablet 85156d7df0344620b71cdb0f128720b6 (DEFAULT_TABLE table=test-table [id=c626264042aa4840b3e8de3196a748b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:31.663803 20072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85156d7df0344620b71cdb0f128720b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:31.663816 19939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85156d7df0344620b71cdb0f128720b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:31.665138 20205 tablet_service.cc:1467] Processing CreateTablet for tablet 85156d7df0344620b71cdb0f128720b6 (DEFAULT_TABLE table=test-table [id=c626264042aa4840b3e8de3196a748b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:31.666057 20205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85156d7df0344620b71cdb0f128720b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:31.669934 20288 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Bootstrap starting.
I20260927 16:28:31.671486 20289 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Bootstrap starting.
I20260927 16:28:31.672057 20288 tablet_bootstrap.cc:654] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:31.672663 20290 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Bootstrap starting.
I20260927 16:28:31.673355 20288 log.cc:824] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:31.673466 20289 tablet_bootstrap.cc:654] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:31.674690 20289 log.cc:824] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:31.674924 20290 tablet_bootstrap.cc:654] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:31.675945 20288 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: No bootstrap required, opened a new log
I20260927 16:28:31.676090 20290 log.cc:824] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:31.676144 20288 ts_tablet_manager.cc:1397] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 16:28:31.676410 20289 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: No bootstrap required, opened a new log
I20260927 16:28:31.676597 20289 ts_tablet_manager.cc:1397] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:31.677918 20290 tablet_bootstrap.cc:492] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: No bootstrap required, opened a new log
I20260927 16:28:31.678117 20290 ts_tablet_manager.cc:1397] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:28:31.679188 20288 raft_consensus.cc:348] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.679438 20288 raft_consensus.cc:374] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:31.679503 20288 raft_consensus.cc:729] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4708f0c669234e3f806208a1d24ee4c0, State: Initialized, Role: FOLLOWER
I20260927 16:28:31.679930 20288 consensus_queue.cc:260] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.680377 20289 raft_consensus.cc:348] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.680629 20138 heartbeater.cc:503] Master 127.15.93.190:32999 was elected leader, sending a full tablet report...
I20260927 16:28:31.680625 20289 raft_consensus.cc:374] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:31.680687 20289 raft_consensus.cc:729] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5766fb90d84638839f779a1e6c9849, State: Initialized, Role: FOLLOWER
I20260927 16:28:31.680861 20288 ts_tablet_manager.cc:1428] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 16:28:31.681064 20289 consensus_queue.cc:260] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.681846 20290 raft_consensus.cc:348] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.682080 20290 raft_consensus.cc:374] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:31.682124 20290 raft_consensus.cc:729] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbf9cb6a3564d4ba43b81fc4972a494, State: Initialized, Role: FOLLOWER
I20260927 16:28:31.682435 20289 ts_tablet_manager.cc:1428] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:28:31.682505 20290 consensus_queue.cc:260] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.682781 20005 heartbeater.cc:503] Master 127.15.93.190:32999 was elected leader, sending a full tablet report...
I20260927 16:28:31.683037 20271 heartbeater.cc:503] Master 127.15.93.190:32999 was elected leader, sending a full tablet report...
I20260927 16:28:31.683337 20290 ts_tablet_manager.cc:1428] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 16:28:31.711338 20006 tablet.cc:2369] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:31.767445 20294 raft_consensus.cc:482] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:31.767644 20294 raft_consensus.cc:504] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.768900 20294 leader_election.cc:288] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbf9cb6a3564d4ba43b81fc4972a494 (127.15.93.131:44381), 4e5766fb90d84638839f779a1e6c9849 (127.15.93.129:41273)
I20260927 16:28:31.772891 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.769212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41273 (local address 127.15.93.130:55443)
0927 16:28:31.769571 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.769584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.769681 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.769909 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.769911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.769924 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:31.770052 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.770058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.770764 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.770766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.771745 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.771758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.772480 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.772484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.772579 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.772603 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.772670 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.772744 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260927 16:28:31.772914 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.769098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44381 (local address 127.15.93.130:35437)
0927 16:28:31.769286 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.769304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:31.769397 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:31.769725 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:31.769728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:31.769753 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:31.769887 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.769894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.770670 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.770673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.771665 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:31.771675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.772503 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.772506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:31.772624 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.772647 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:28:31.772746 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:28:31.772808 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":1}
I20260927 16:28:31.772914 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.769288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55443 (local address 127.15.93.129:41273)
0927 16:28:31.769416 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.769421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.769683 (+ 262us) server_negotiation.cc:407] Connection header received
0927 16:28:31.769761 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.769764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.769808 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.769893 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:31.770155 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.770660 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.771876 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.772362 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.772486 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.772540 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.772714 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.772791 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:31.772958 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.769231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:35437 (local address 127.15.93.131:44381)
0927 16:28:31.769361 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.769365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.769381 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:31.769531 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.769535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.769597 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.769685 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:31.770000 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.770555 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.771796 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.772371 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.772501 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.772551 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.772775 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.772843 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:28:31.773478 20225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85156d7df0344620b71cdb0f128720b6" candidate_uuid: "4708f0c669234e3f806208a1d24ee4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" is_pre_election: true
I20260927 16:28:31.773754 20225 raft_consensus.cc:2454] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4708f0c669234e3f806208a1d24ee4c0 in term 0.
I20260927 16:28:31.774044 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85156d7df0344620b71cdb0f128720b6" candidate_uuid: "4708f0c669234e3f806208a1d24ee4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5766fb90d84638839f779a1e6c9849" is_pre_election: true
I20260927 16:28:31.774262 20027 leader_election.cc:302] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbf9cb6a3564d4ba43b81fc4972a494, 4708f0c669234e3f806208a1d24ee4c0; no voters:
I20260927 16:28:31.774333 19959 raft_consensus.cc:2454] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4708f0c669234e3f806208a1d24ee4c0 in term 0.
I20260927 16:28:31.774537 20294 raft_consensus.cc:2781] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:31.774631 20294 raft_consensus.cc:482] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:31.774689 20294 raft_consensus.cc:3037] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:31.775763 20294 raft_consensus.cc:504] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.776077 20294 leader_election.cc:288] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbf9cb6a3564d4ba43b81fc4972a494 (127.15.93.131:44381), 4e5766fb90d84638839f779a1e6c9849 (127.15.93.129:41273)
I20260927 16:28:31.776419 20225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85156d7df0344620b71cdb0f128720b6" candidate_uuid: "4708f0c669234e3f806208a1d24ee4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494"
I20260927 16:28:31.776451 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85156d7df0344620b71cdb0f128720b6" candidate_uuid: "4708f0c669234e3f806208a1d24ee4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5766fb90d84638839f779a1e6c9849"
I20260927 16:28:31.776549 19959 raft_consensus.cc:3037] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:31.776546 20225 raft_consensus.cc:3037] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:31.777770 19959 raft_consensus.cc:2454] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4708f0c669234e3f806208a1d24ee4c0 in term 1.
I20260927 16:28:31.777770 20225 raft_consensus.cc:2454] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4708f0c669234e3f806208a1d24ee4c0 in term 1.
I20260927 16:28:31.778088 20027 leader_election.cc:302] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cbf9cb6a3564d4ba43b81fc4972a494, 4708f0c669234e3f806208a1d24ee4c0, 4e5766fb90d84638839f779a1e6c9849; no voters:
I20260927 16:28:31.778291 20294 raft_consensus.cc:2781] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:31.778473 20294 raft_consensus.cc:686] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4708f0c669234e3f806208a1d24ee4c0, State: Running, Role: LEADER
I20260927 16:28:31.778788 20294 consensus_queue.cc:237] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } }
I20260927 16:28:31.781801 19813 catalog_manager.cc:5495] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4708f0c669234e3f806208a1d24ee4c0 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "4708f0c669234e3f806208a1d24ee4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4708f0c669234e3f806208a1d24ee4c0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:31.808900 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.805698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.15.93.130:33973)
0927 16:28:31.805818 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:31.805822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:31.805835 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:28:31.805927 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:31.805929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:31.805983 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:31.806070 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:31.806325 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.806812 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.807690 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:31.807887 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:31.807958 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:31.808209 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:31.808597 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:31.808685 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:31.808736 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:28:31.808781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:28:31.814316 19959 raft_consensus.cc:1261] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 1 FOLLOWER]: Refusing update from remote peer 4708f0c669234e3f806208a1d24ee4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:31.814316 20225 raft_consensus.cc:1261] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 1 FOLLOWER]: Refusing update from remote peer 4708f0c669234e3f806208a1d24ee4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:31.815056 20298 consensus_queue.cc:1035] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5766fb90d84638839f779a1e6c9849" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:31.815241 20294 consensus_queue.cc:1035] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbf9cb6a3564d4ba43b81fc4972a494" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:31.823021 20303 mvcc.cc:205] Tried to move back new op lower bound from 7333996592384696320 to 7333996592248590336. Current Snapshot: MvccSnapshot[applied={T|T < 7333996592384696320}]
I20260927 16:28:31.831571 19815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:31.831745 19815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:31.834092 19815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4b37750bb3e437fa211382fa7bf8c47: Sending DeleteTablet for 3 replicas of tablet 85156d7df0344620b71cdb0f128720b6
I20260927 16:28:31.834892 20205 tablet_service.cc:1514] Processing DeleteTablet for tablet 85156d7df0344620b71cdb0f128720b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:31 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:41242
I20260927 16:28:31.834883 19939 tablet_service.cc:1514] Processing DeleteTablet for tablet 85156d7df0344620b71cdb0f128720b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:31 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58912
I20260927 16:28:31.834894 20072 tablet_service.cc:1514] Processing DeleteTablet for tablet 85156d7df0344620b71cdb0f128720b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:31 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:52248
I20260927 16:28:31.835356 20313 tablet_replica.cc:330] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: stopping tablet replica
I20260927 16:28:31.835614 20311 tablet_replica.cc:330] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849: stopping tablet replica
I20260927 16:28:31.835682 20313 raft_consensus.cc:2229] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:31.835878 20311 raft_consensus.cc:2229] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:31.836048 20313 raft_consensus.cc:2258] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:31.836180 20311 raft_consensus.cc:2258] T 85156d7df0344620b71cdb0f128720b6 P 4e5766fb90d84638839f779a1e6c9849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:31.836298 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19877
I20260927 16:28:31.838253 20313 ts_tablet_manager.cc:1905] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:31.838640 20312 tablet_replica.cc:330] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: stopping tablet replica
I20260927 16:28:31.838923 20312 raft_consensus.cc:2229] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:31.839259 20312 raft_consensus.cc:2258] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:31.841151 20313 ts_tablet_manager.cc:1918] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:31.841245 20313 log.cc:1196] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/85156d7df0344620b71cdb0f128720b6
I20260927 16:28:31.841552 20313 ts_tablet_manager.cc:1939] T 85156d7df0344620b71cdb0f128720b6 P 4708f0c669234e3f806208a1d24ee4c0: Deleting consensus metadata
I20260927 16:28:31.842684 19801 catalog_manager.cc:4841] TS 4708f0c669234e3f806208a1d24ee4c0 (127.15.93.130:33973): tablet 85156d7df0344620b71cdb0f128720b6 (table test-table [id=c626264042aa4840b3e8de3196a748b1]) successfully deleted
W20260927 16:28:31.843889 19802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:41273 (error 108)
I20260927 16:28:31.844282 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20010
I20260927 16:28:31.846247 20312 ts_tablet_manager.cc:1905] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:31.848392 20312 ts_tablet_manager.cc:1918] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:31.848469 20312 log.cc:1196] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/85156d7df0344620b71cdb0f128720b6
I20260927 16:28:31.848754 20312 ts_tablet_manager.cc:1939] T 85156d7df0344620b71cdb0f128720b6 P 1cbf9cb6a3564d4ba43b81fc4972a494: Deleting consensus metadata
I20260927 16:28:31.849483 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20143
I20260927 16:28:31.850296 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:31.849760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41273 (local address 127.0.0.1:58918)
0927 16:28:31.850170 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:31.850225 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
W20260927 16:28:31.850517 19802 catalog_manager.cc:4584] TS 4e5766fb90d84638839f779a1e6c9849 (127.15.93.129:41273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85156d7df0344620b71cdb0f128720b6: Network error: Client connection negotiation failed: client connection to 127.15.93.129:41273: connect: Connection refused (error 111)
I20260927 16:28:31.855366 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 19784
2026-09-27T16:28:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2272 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T16:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:31Z Disabled control of system clock
I20260927 16:28:31.885255 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:35679
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:35679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:28:31.990087 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:31.990320 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:31.990355 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:31.994164 20318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:31.994213 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:31.994228 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:31.994246 20318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:31.994259 20318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:31.998761 20318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:35679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:31.999722 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:32.000959 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:32.008929 20323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.008961 20324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.008948 20326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.009176 20318 server_base.cc:1034] running on GCE node
I20260927 16:28:32.009722 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:32.010670 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:32.011857 20318 hybrid_clock.cc:648] HybridClock initialized: now 1790526512011826 us; error 44 us; skew 500 ppm
I20260927 16:28:32.013516 20318 webserver.cc:460] Webserver started at http://127.15.93.190:42749/ using document root <none> and password file <none>
I20260927 16:28:32.014063 20318 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:32.014160 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:32.014380 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:32.016398 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "615a1b7998b945deaeb0381e84ac7a7a"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "b893bd67e28f82a531ef6b49d5a23662"
server_key_iv: "ac39d26be427db73db1e849c9d8e9902"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.016925 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "615a1b7998b945deaeb0381e84ac7a7a"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "b893bd67e28f82a531ef6b49d5a23662"
server_key_iv: "ac39d26be427db73db1e849c9d8e9902"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.020364 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:32.022644 20332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.023969 20318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:32.024081 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "615a1b7998b945deaeb0381e84ac7a7a"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "b893bd67e28f82a531ef6b49d5a23662"
server_key_iv: "ac39d26be427db73db1e849c9d8e9902"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.024164 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:32.028448 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:32.029017 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:32.029150 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:32.035732 20318 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:35679
I20260927 16:28:32.035742 20384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:35679 every 8 connection(s)
I20260927 16:28:32.036612 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:32.039554 20385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:32.039876 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20318
I20260927 16:28:32.039968 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:32.040271 15734 external_mini_cluster.cc:1428] Setting key 92b9974dc8a5a88f1bc54163ff881c48
I20260927 16:28:32.046030 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.041612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.15.93.190:35679)
0927 16:28:32.042256 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.042276 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.042318 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:28:32.043141 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.043163 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.043611 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.043966 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.044664 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.044669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.044683 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.044694 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.044756 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.044777 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.044915 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.044987 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.045220 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260927 16:28:32.046123 20385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: Bootstrap starting.
I20260927 16:28:32.048394 20385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:32.049063 20385 log.cc:824] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:32.050724 20385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: No bootstrap required, opened a new log
I20260927 16:28:32.054033 20385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } }
I20260927 16:28:32.054237 20385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:32.054270 20385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615a1b7998b945deaeb0381e84ac7a7a, State: Initialized, Role: FOLLOWER
I20260927 16:28:32.054607 20385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } }
I20260927 16:28:32.054718 20385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:32.054762 20385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:32.054822 20385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:32.055641 20385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } }
I20260927 16:28:32.055900 20385 leader_election.cc:302] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615a1b7998b945deaeb0381e84ac7a7a; no voters:
I20260927 16:28:32.056159 20385 leader_election.cc:288] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:32.056259 20390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:32.056473 20390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [term 1 LEADER]: Becoming Leader. State: Replica: 615a1b7998b945deaeb0381e84ac7a7a, State: Running, Role: LEADER
I20260927 16:28:32.056764 20390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } }
I20260927 16:28:32.057331 20385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:32.058233 20391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615a1b7998b945deaeb0381e84ac7a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } } }
I20260927 16:28:32.058211 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615a1b7998b945deaeb0381e84ac7a7a. Latest consensus state: current_term: 1 leader_uuid: "615a1b7998b945deaeb0381e84ac7a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a1b7998b945deaeb0381e84ac7a7a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35679 } } }
I20260927 16:28:32.058457 20391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:32.058458 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:32.058883 20399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:32.061980 20399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:32.067325 20399 catalog_manager.cc:1347] Generated new cluster ID: 7be107f9455e41a5b86a34c08043720c
I20260927 16:28:32.067433 20399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:32.100826 20399 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:32.101162 20399 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:32.108212 20399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: Generated new TSK 0
I20260927 16:28:32.108952 20399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:28:32.124794 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--builtin_ntp_servers=127.15.93.148:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:28:32.254750 20409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:32.254994 20409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:32.255033 20409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:32.259231 20409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:32.259279 20409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:32.259333 20409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:32.264091 20409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:32.265064 20409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:32.266331 20409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:32.275707 20414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.275723 20409 server_base.cc:1034] running on GCE node
W20260927 16:28:32.275740 20417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.275717 20415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.276312 20409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:32.276852 20409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:32.278002 20409 hybrid_clock.cc:648] HybridClock initialized: now 1790526512277989 us; error 38 us; skew 500 ppm
I20260927 16:28:32.279755 20409 webserver.cc:460] Webserver started at http://127.15.93.129:44297/ using document root <none> and password file <none>
I20260927 16:28:32.280360 20409 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:32.280444 20409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:32.280679 20409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:32.282781 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "9416c4063132400bbab23f6e33d49f1e"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "48b4b4250434d955f07a074e3627cafc"
server_key_iv: "eda7089e22b56db30f587218f7793591"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.283273 20409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "9416c4063132400bbab23f6e33d49f1e"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "48b4b4250434d955f07a074e3627cafc"
server_key_iv: "eda7089e22b56db30f587218f7793591"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.286898 20409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:28:32.289268 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.290408 20409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:32.290526 20409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "9416c4063132400bbab23f6e33d49f1e"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "48b4b4250434d955f07a074e3627cafc"
server_key_iv: "eda7089e22b56db30f587218f7793591"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.290643 20409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:32.294996 20409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:32.295622 20409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:32.295800 20409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:32.296304 20409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:32.297168 20409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:32.297212 20409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.297276 20409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:32.297305 20409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.305333 20409 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:36493
I20260927 16:28:32.305413 20536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:36493 every 8 connection(s)
I20260927 16:28:32.306218 20409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:32.311223 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20409
I20260927 16:28:32.311354 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:32.311666 15734 external_mini_cluster.cc:1428] Setting key 629e9e0f2e1ef37fda502d641c0de0d6
I20260927 16:28:32.312261 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.307984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:55737 (local address 127.15.93.190:35679)
0927 16:28:32.308117 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.308121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.309120 (+ 999us) server_negotiation.cc:407] Connection header received
0927 16:28:32.310456 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.310460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.310505 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.310613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.311794 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.311797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.311800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.311819 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.311837 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.311845 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.311945 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.312083 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.312129 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:32.313185 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.308351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35679 (local address 127.15.93.129:55737)
0927 16:28:32.308979 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.309017 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.310240 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.310782 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.310789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.310876 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.311551 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.311552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.311591 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.311656 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.311930 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.311938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.311944 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.312117 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.312354 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260927 16:28:32.313756 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--builtin_ntp_servers=127.15.93.148:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 16:28:32.314391 20537 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35679
I20260927 16:28:32.314610 20537 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:32.315014 20537 heartbeater.cc:511] Master 127.15.93.190:35679 requested a full tablet report, sending...
I20260927 16:28:32.316558 20349 ts_manager.cc:194] Registered new tserver with Master: 9416c4063132400bbab23f6e33d49f1e (127.15.93.129:36493)
W20260927 16:28:32.421737 20540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:32.421993 20540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:32.422060 20540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:32.426030 20540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:32.426102 20540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:32.426189 20540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:32.430799 20540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:32.431823 20540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:32.433102 20540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:32.442147 20548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.442126 20545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.442121 20546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.442562 20540 server_base.cc:1034] running on GCE node
I20260927 16:28:32.442938 20540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:32.443534 20540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:32.444725 20540 hybrid_clock.cc:648] HybridClock initialized: now 1790526512444709 us; error 25 us; skew 500 ppm
I20260927 16:28:32.446391 20540 webserver.cc:460] Webserver started at http://127.15.93.130:33355/ using document root <none> and password file <none>
I20260927 16:28:32.446934 20540 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:32.447021 20540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:32.447242 20540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:32.449261 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c7611649f8d8bc69dd6fe675b0842cb9"
server_key_iv: "c4ba20c5dc25b0674aa002dd8d4f4e05"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.449807 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c7611649f8d8bc69dd6fe675b0842cb9"
server_key_iv: "c4ba20c5dc25b0674aa002dd8d4f4e05"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.453327 20540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:32.455688 20554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.456848 20540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:32.456995 20540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c7611649f8d8bc69dd6fe675b0842cb9"
server_key_iv: "c4ba20c5dc25b0674aa002dd8d4f4e05"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.457093 20540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:32.461485 20540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:32.461992 20540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:32.462118 20540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:32.462570 20540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:32.463349 20540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:32.463428 20540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.463465 20540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:32.463486 20540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.471316 20540 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:46289
I20260927 16:28:32.471412 20667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:46289 every 8 connection(s)
I20260927 16:28:32.472333 20540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:32.477819 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.473981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41809 (local address 127.15.93.190:35679)
0927 16:28:32.474131 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.474136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.475108 (+ 972us) server_negotiation.cc:407] Connection header received
0927 16:28:32.476294 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.476297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.476331 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.476406 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.477454 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.477456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.477458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.477466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.477481 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.477487 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.477552 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.477683 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.477717 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:28:32.478609 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.474369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35679 (local address 127.15.93.130:41809)
0927 16:28:32.474969 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.475005 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.476118 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.476578 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.476586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.476686 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.477241 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.477242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.477278 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.477340 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.477554 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.477560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.477564 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.477730 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.477943 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260927 16:28:32.479700 20668 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35679
I20260927 16:28:32.479903 20668 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:32.480273 20668 heartbeater.cc:511] Master 127.15.93.190:35679 requested a full tablet report, sending...
I20260927 16:28:32.480883 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20540
I20260927 16:28:32.481015 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:32.481218 20349 ts_manager.cc:194] Registered new tserver with Master: 62fa6522d9b6419ab5c92f3ddc5fb7d0 (127.15.93.130:46289)
I20260927 16:28:32.481317 15734 external_mini_cluster.cc:1428] Setting key ed4b3c63d2f29643f745cc5f9aae0693
I20260927 16:28:32.483089 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--builtin_ntp_servers=127.15.93.148:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 16:28:32.588995 20671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:32.589217 20671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:32.589254 20671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:32.593142 20671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:32.593183 20671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:32.593281 20671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:32.597903 20671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:32.598850 20671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:32.600126 20671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:32.609297 20677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.609305 20676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.609287 20679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.609695 20671 server_base.cc:1034] running on GCE node
I20260927 16:28:32.610040 20671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:32.610635 20671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:32.611783 20671 hybrid_clock.cc:648] HybridClock initialized: now 1790526512611753 us; error 43 us; skew 500 ppm
I20260927 16:28:32.613453 20671 webserver.cc:460] Webserver started at http://127.15.93.131:37827/ using document root <none> and password file <none>
I20260927 16:28:32.613961 20671 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:32.614017 20671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:32.614183 20671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:32.616221 20671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c402fdbf70103ca4d17b207b89c2220d"
server_key_iv: "1d0bc475bf66c4039a7dfe3db2f896c3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.616714 20671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c402fdbf70103ca4d17b207b89c2220d"
server_key_iv: "1d0bc475bf66c4039a7dfe3db2f896c3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.620142 20671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:32.622489 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.623698 20671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:32.623792 20671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "c402fdbf70103ca4d17b207b89c2220d"
server_key_iv: "1d0bc475bf66c4039a7dfe3db2f896c3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.623874 20671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:32.628621 20671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:32.629163 20671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:32.629289 20671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:32.629732 20671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:32.630523 20671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:32.630568 20671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.630602 20671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:32.630623 20671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.638859 20671 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:33051
I20260927 16:28:32.638911 20798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:33051 every 8 connection(s)
I20260927 16:28:32.639750 20671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:32.645404 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.641426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:59271 (local address 127.15.93.190:35679)
0927 16:28:32.641586 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.641590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.642607 (+ 1017us) server_negotiation.cc:407] Connection header received
0927 16:28:32.643884 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.643886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.643922 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.643996 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.645043 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.645045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.645047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.645055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.645070 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.645076 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.645146 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.645270 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.645305 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:28:32.646198 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.641804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35679 (local address 127.15.93.131:59271)
0927 16:28:32.642463 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.642500 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.643702 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.644153 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.644159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.644241 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.644810 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.644811 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.644849 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.644913 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.645155 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.645161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.645164 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.645285 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.645481 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260927 16:28:32.647233 20799 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35679
I20260927 16:28:32.647495 20799 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:32.647859 20799 heartbeater.cc:511] Master 127.15.93.190:35679 requested a full tablet report, sending...
I20260927 16:28:32.648469 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20671
I20260927 16:28:32.648592 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:32.648875 15734 external_mini_cluster.cc:1428] Setting key ee28d7955a3a168efb510a51a3e80827
I20260927 16:28:32.648918 20349 ts_manager.cc:194] Registered new tserver with Master: 28cfdc4ed6234a8fabd818ac3bf4fa3f (127.15.93.131:33051)
I20260927 16:28:32.650946 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:28:32.654109 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.652718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.15.93.190:35679)
0927 16:28:32.652854 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.652859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.652970 (+ 111us) server_negotiation.cc:407] Connection header received
0927 16:28:32.653043 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.653046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.653074 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.653136 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.653793 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.653796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.653798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.653806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.653821 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.653827 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.653890 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.653927 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.653954 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:28:32.656637 20349 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:32.659641 20349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:32.670377 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.667357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:46289 (local address 127.0.0.1:47402)
0927 16:28:32.667923 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.667935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.667992 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.669239 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.669241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.669256 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.669718 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.669720 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.669729 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.669749 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.670038 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.670047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.670052 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.670185 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.670253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":95,"threads_started":1}
I20260927 16:28:32.670473 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.668397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.15.93.130:46289)
0927 16:28:32.668867 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.668878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.668898 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:28:32.668943 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.668948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.669112 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.669255 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.669891 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.669896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.669903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.669912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.669939 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.669955 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.670064 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.670259 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.670329 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":111,"threads_started":1}
I20260927 16:28:32.670615 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.667661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:33051 (local address 127.0.0.1:60634)
0927 16:28:32.668380 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.668393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.668486 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.668877 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.668884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.668954 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.669455 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.669457 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.669476 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.669543 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.670382 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.670386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.670389 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.670447 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.670493 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":85,"threads_started":1}
I20260927 16:28:32.670827 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.667917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.15.93.129:36493)
0927 16:28:32.668275 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.668286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.668311 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:28:32.668365 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.668369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.668550 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.669349 (+ 799us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.670204 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.670209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.670216 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.670229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.670259 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.670281 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.670540 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.670581 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.670660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260927 16:28:32.670981 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.667267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:36493 (local address 127.0.0.1:60156)
0927 16:28:32.667579 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.667630 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.667923 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.669123 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.669126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.669147 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.669568 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.669571 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.669583 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.669615 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.670382 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.670386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.670389 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.670447 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.670874 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":373,"thread_start_us":107,"threads_started":1}
I20260927 16:28:32.671206 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.667994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.15.93.131:33051)
0927 16:28:32.668275 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.668286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.668482 (+ 196us) server_negotiation.cc:407] Connection header received
0927 16:28:32.668570 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.668574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.668728 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.668863 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.670059 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.670167 (+ 108us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.670181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.670195 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.670240 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.670270 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.670948 (+ 678us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.670992 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.671055 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260927 16:28:32.673491 20602 tablet_service.cc:1467] Processing CreateTablet for tablet 79469dd4a2c742818ce80660789aafc9 (DEFAULT_TABLE table=test-table [id=6195f39f33b049a98003dbf90c84de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:32.673872 20471 tablet_service.cc:1467] Processing CreateTablet for tablet 79469dd4a2c742818ce80660789aafc9 (DEFAULT_TABLE table=test-table [id=6195f39f33b049a98003dbf90c84de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:32.674606 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79469dd4a2c742818ce80660789aafc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:32.674832 20471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79469dd4a2c742818ce80660789aafc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:32.675689 20733 tablet_service.cc:1467] Processing CreateTablet for tablet 79469dd4a2c742818ce80660789aafc9 (DEFAULT_TABLE table=test-table [id=6195f39f33b049a98003dbf90c84de30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:32.676432 20733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79469dd4a2c742818ce80660789aafc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:32.681651 20817 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: Bootstrap starting.
I20260927 16:28:32.682052 20816 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Bootstrap starting.
I20260927 16:28:32.682966 20818 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Bootstrap starting.
I20260927 16:28:32.684310 20817 tablet_bootstrap.cc:654] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:32.684376 20816 tablet_bootstrap.cc:654] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:32.684835 20818 tablet_bootstrap.cc:654] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:32.685070 20817 log.cc:824] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:32.685230 20816 log.cc:824] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:32.685563 20818 log.cc:824] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:32.686900 20816 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: No bootstrap required, opened a new log
I20260927 16:28:32.686992 20817 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: No bootstrap required, opened a new log
I20260927 16:28:32.687119 20816 ts_tablet_manager.cc:1397] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:32.687189 20817 ts_tablet_manager.cc:1397] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 16:28:32.687294 20818 tablet_bootstrap.cc:492] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: No bootstrap required, opened a new log
I20260927 16:28:32.687477 20818 ts_tablet_manager.cc:1397] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:32.690653 20816 raft_consensus.cc:348] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.690888 20816 raft_consensus.cc:374] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:32.690958 20816 raft_consensus.cc:729] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fa6522d9b6419ab5c92f3ddc5fb7d0, State: Initialized, Role: FOLLOWER
I20260927 16:28:32.691407 20816 consensus_queue.cc:260] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.691615 20818 raft_consensus.cc:348] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.691615 20817 raft_consensus.cc:348] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.691838 20817 raft_consensus.cc:374] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:32.691900 20818 raft_consensus.cc:374] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:32.691922 20817 raft_consensus.cc:729] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9416c4063132400bbab23f6e33d49f1e, State: Initialized, Role: FOLLOWER
I20260927 16:28:32.691988 20818 raft_consensus.cc:729] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cfdc4ed6234a8fabd818ac3bf4fa3f, State: Initialized, Role: FOLLOWER
I20260927 16:28:32.692026 20668 heartbeater.cc:503] Master 127.15.93.190:35679 was elected leader, sending a full tablet report...
I20260927 16:28:32.692375 20817 consensus_queue.cc:260] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.692375 20818 consensus_queue.cc:260] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.693068 20537 heartbeater.cc:503] Master 127.15.93.190:35679 was elected leader, sending a full tablet report...
I20260927 16:28:32.693293 20817 ts_tablet_manager.cc:1428] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:28:32.693293 20818 ts_tablet_manager.cc:1428] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:28:32.693565 20799 heartbeater.cc:503] Master 127.15.93.190:35679 was elected leader, sending a full tablet report...
I20260927 16:28:32.693912 20816 ts_tablet_manager.cc:1428] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 16:28:32.723258 20669 tablet.cc:2369] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:32.735474 20822 raft_consensus.cc:482] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:32.735639 20822 raft_consensus.cc:504] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.736744 20822 leader_election.cc:288] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9416c4063132400bbab23f6e33d49f1e (127.15.93.129:36493), 28cfdc4ed6234a8fabd818ac3bf4fa3f (127.15.93.131:33051)
I20260927 16:28:32.738577 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.736925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:36493 (local address 127.15.93.130:47127)
0927 16:28:32.737129 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.737146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.737225 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.737478 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.737481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.737496 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.737991 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.737993 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.738007 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.738041 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.738308 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.738311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.738314 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.738399 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.738441 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 16:28:32.738610 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.737050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:47127 (local address 127.15.93.129:36493)
0927 16:28:32.737199 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.737203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.737219 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:32.737324 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.737329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.737371 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.737460 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.738167 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.738171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.738174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.738186 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.738206 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.738215 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.738302 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.738408 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.738465 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:28:32.738727 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.737120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:59547 (local address 127.15.93.131:33051)
0927 16:28:32.737239 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.737246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.737462 (+ 216us) server_negotiation.cc:407] Connection header received
0927 16:28:32.737597 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.737602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.737664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.737752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.738328 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.738331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.738336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.738351 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.738378 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.738390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.738470 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.738557 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.738596 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:32.738746 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.736996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:33051 (local address 127.15.93.130:59547)
0927 16:28:32.737369 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.737380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:32.737493 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:32.737770 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:32.737784 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:32.737800 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:32.738163 (+ 363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:28:32.738165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:32.738175 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:28:32.738199 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:32.738471 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:32.738474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:32.738477 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:28:32.738552 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:28:32.738584 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260927 16:28:32.739310 20491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79469dd4a2c742818ce80660789aafc9" candidate_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416c4063132400bbab23f6e33d49f1e" is_pre_election: true
I20260927 16:28:32.739331 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79469dd4a2c742818ce80660789aafc9" candidate_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" is_pre_election: true
I20260927 16:28:32.739663 20753 raft_consensus.cc:2454] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62fa6522d9b6419ab5c92f3ddc5fb7d0 in term 0.
I20260927 16:28:32.739665 20491 raft_consensus.cc:2454] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62fa6522d9b6419ab5c92f3ddc5fb7d0 in term 0.
I20260927 16:28:32.740105 20555 leader_election.cc:302] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cfdc4ed6234a8fabd818ac3bf4fa3f, 62fa6522d9b6419ab5c92f3ddc5fb7d0; no voters:
I20260927 16:28:32.740355 20822 raft_consensus.cc:2781] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:32.740434 20822 raft_consensus.cc:482] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:32.740476 20822 raft_consensus.cc:3037] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:32.741482 20822 raft_consensus.cc:504] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.741802 20822 leader_election.cc:288] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [CANDIDATE]: Term 1 election: Requested vote from peers 9416c4063132400bbab23f6e33d49f1e (127.15.93.129:36493), 28cfdc4ed6234a8fabd818ac3bf4fa3f (127.15.93.131:33051)
I20260927 16:28:32.742117 20753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79469dd4a2c742818ce80660789aafc9" candidate_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f"
I20260927 16:28:32.742125 20491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79469dd4a2c742818ce80660789aafc9" candidate_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9416c4063132400bbab23f6e33d49f1e"
I20260927 16:28:32.742242 20753 raft_consensus.cc:3037] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:32.742242 20491 raft_consensus.cc:3037] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:32.743512 20491 raft_consensus.cc:2454] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62fa6522d9b6419ab5c92f3ddc5fb7d0 in term 1.
I20260927 16:28:32.743512 20753 raft_consensus.cc:2454] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62fa6522d9b6419ab5c92f3ddc5fb7d0 in term 1.
I20260927 16:28:32.743782 20557 leader_election.cc:302] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fa6522d9b6419ab5c92f3ddc5fb7d0, 9416c4063132400bbab23f6e33d49f1e; no voters:
I20260927 16:28:32.743937 20822 raft_consensus.cc:2781] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:32.744148 20822 raft_consensus.cc:686] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 1 LEADER]: Becoming Leader. State: Replica: 62fa6522d9b6419ab5c92f3ddc5fb7d0, State: Running, Role: LEADER
I20260927 16:28:32.744477 20822 consensus_queue.cc:237] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } }
I20260927 16:28:32.747707 20347 catalog_manager.cc:5495] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62fa6522d9b6419ab5c92f3ddc5fb7d0 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fa6522d9b6419ab5c92f3ddc5fb7d0" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:32.773897 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.772305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.15.93.130:46289)
0927 16:28:32.772459 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:32.772464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:32.772479 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:32.772604 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:32.772607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:32.772674 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:32.772768 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:32.773491 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:32.773506 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:32.773508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:32.773519 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:32.773544 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:32.773552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:32.773645 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:32.773724 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 16:28:32.773761 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:28:32.779562 20753 raft_consensus.cc:1261] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 1 FOLLOWER]: Refusing update from remote peer 62fa6522d9b6419ab5c92f3ddc5fb7d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:32.779562 20491 raft_consensus.cc:1261] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 1 FOLLOWER]: Refusing update from remote peer 62fa6522d9b6419ab5c92f3ddc5fb7d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:32.780264 20822 consensus_queue.cc:1035] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9416c4063132400bbab23f6e33d49f1e" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:32.780483 20826 consensus_queue.cc:1035] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28cfdc4ed6234a8fabd818ac3bf4fa3f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:32.797513 20348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:32.797696 20348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:32.799990 20348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 615a1b7998b945deaeb0381e84ac7a7a: Sending DeleteTablet for 3 replicas of tablet 79469dd4a2c742818ce80660789aafc9
I20260927 16:28:32.800688 20471 tablet_service.cc:1514] Processing DeleteTablet for tablet 79469dd4a2c742818ce80660789aafc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:32 UTC) from {username='slave'} at 127.0.0.1:60156
I20260927 16:28:32.800736 20733 tablet_service.cc:1514] Processing DeleteTablet for tablet 79469dd4a2c742818ce80660789aafc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:32 UTC) from {username='slave'} at 127.0.0.1:60634
I20260927 16:28:32.800776 20602 tablet_service.cc:1514] Processing DeleteTablet for tablet 79469dd4a2c742818ce80660789aafc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:32 UTC) from {username='slave'} at 127.0.0.1:47402
I20260927 16:28:32.801270 20841 tablet_replica.cc:330] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: stopping tablet replica
I20260927 16:28:32.801280 20839 tablet_replica.cc:330] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: stopping tablet replica
I20260927 16:28:32.801560 20839 raft_consensus.cc:2229] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:32.801807 20840 tablet_replica.cc:330] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e: stopping tablet replica
I20260927 16:28:32.801939 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20409
I20260927 16:28:32.802139 20840 raft_consensus.cc:2229] T 79469dd4a2c742818ce80660789aafc9 P 9416c4063132400bbab23f6e33d49f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:32.802646 20839 raft_consensus.cc:2258] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:32.804207 20839 ts_tablet_manager.cc:1905] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:32.806669 20839 ts_tablet_manager.cc:1918] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:32.806782 20839 log.cc:1196] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/79469dd4a2c742818ce80660789aafc9
I20260927 16:28:32.807065 20839 ts_tablet_manager.cc:1939] T 79469dd4a2c742818ce80660789aafc9 P 28cfdc4ed6234a8fabd818ac3bf4fa3f: Deleting consensus metadata
I20260927 16:28:32.808255 20333 catalog_manager.cc:4841] TS 28cfdc4ed6234a8fabd818ac3bf4fa3f (127.15.93.131:33051): tablet 79469dd4a2c742818ce80660789aafc9 (table test-table [id=6195f39f33b049a98003dbf90c84de30]) successfully deleted
I20260927 16:28:32.808629 20841 raft_consensus.cc:2229] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:32.808971 20841 raft_consensus.cc:2258] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:28:32.809860 20335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:36493 (error 108)
I20260927 16:28:32.810326 20841 ts_tablet_manager.cc:1905] T 79469dd4a2c742818ce80660789aafc9 P 62fa6522d9b6419ab5c92f3ddc5fb7d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:32.810823 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.810524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:36493 (local address 127.0.0.1:60168)
0927 16:28:32.810655 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.810728 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:28:32.810847 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20540
W20260927 16:28:32.813449 20335 catalog_manager.cc:4584] TS 9416c4063132400bbab23f6e33d49f1e (127.15.93.129:36493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79469dd4a2c742818ce80660789aafc9: Network error: Client connection negotiation failed: client connection to 127.15.93.129:36493: connect: Connection refused (error 111)
I20260927 16:28:32.818059 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20671
I20260927 16:28:32.818292 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:32.818033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:46289 (local address 127.0.0.1:47420)
0927 16:28:32.818146 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:32.818204 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.130:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:28:32.823658 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20318
2026-09-27T16:28:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (967 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 16:28:32.845206 15734 mini_oidc.cc:152] Starting JWKS server
I20260927 16:28:32.846680 15734 webserver.cc:460] Webserver started at https://127.0.0.1:35097/ using document root <none> and password file <none>
I20260927 16:28:32.846812 15734 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:28:32.847020 15734 webserver.cc:460] Webserver started at http://127.0.0.1:39375/ using document root <none> and password file <none>
2026-09-27T16:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:32Z Disabled control of system clock
I20260927 16:28:32.855510 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38643
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:32831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38643
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/testchainca.cert with env {}
W20260927 16:28:32.961473 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:32.961756 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:32.961829 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:32.965785 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:32.965869 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:32.965898 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:32.965938 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:32.965974 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:28:32.966013 20848 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:28:32.966048 20848 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35097/jwks.json
I20260927 16:28:32.971333 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:32831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:32.972349 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:32.973325 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:32.981557 20853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:32.981554 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.981597 20848 server_base.cc:1034] running on GCE node
W20260927 16:28:32.981554 20854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:32.982273 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:32.983340 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:32.984555 20848 hybrid_clock.cc:648] HybridClock initialized: now 1790526512984526 us; error 41 us; skew 500 ppm
I20260927 16:28:32.986214 20848 webserver.cc:460] Webserver started at http://127.15.93.190:42041/ using document root <none> and password file <none>
I20260927 16:28:32.986711 20848 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:32.986765 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:32.986912 20848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:32.988838 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "9115993726044871884e45ac75cbf93f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "41e44d742e4f5e913f4a0b3b5df45295"
server_key_iv: "478bf7c7387dd15c8200bcb6a8cecde3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.989331 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "9115993726044871884e45ac75cbf93f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "41e44d742e4f5e913f4a0b3b5df45295"
server_key_iv: "478bf7c7387dd15c8200bcb6a8cecde3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.992705 20848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:28:32.995030 20862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:32.996289 20848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:28:32.996382 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "9115993726044871884e45ac75cbf93f"
format_stamp: "Formatted at 2026-09-27 16:28:32 on dist-test-slave-grzf"
server_key: "41e44d742e4f5e913f4a0b3b5df45295"
server_key_iv: "478bf7c7387dd15c8200bcb6a8cecde3"
server_key_version: "encryptionkey@0"
I20260927 16:28:32.996464 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:33.022324 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:33.023007 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:33.023209 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:33.029744 20848 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38643
I20260927 16:28:33.029750 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38643 every 8 connection(s)
I20260927 16:28:33.030668 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:33.033663 20915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:33.038802 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: Bootstrap starting.
I20260927 16:28:33.040740 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20848
I20260927 16:28:33.040870 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:33.041148 15734 external_mini_cluster.cc:1428] Setting key 6bce675e046574bb1560211177de78bf
I20260927 16:28:33.041347 20915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:33.042181 20915 log.cc:824] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:33.044106 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: No bootstrap required, opened a new log
I20260927 16:28:33.047488 20915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } }
I20260927 16:28:33.047760 20915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:33.047830 20915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9115993726044871884e45ac75cbf93f, State: Initialized, Role: FOLLOWER
I20260927 16:28:33.048374 20915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } }
I20260927 16:28:33.048549 20915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:33.048623 20915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:33.048715 20915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:33.049777 20915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } }
I20260927 16:28:33.050110 20915 leader_election.cc:302] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9115993726044871884e45ac75cbf93f; no voters:
I20260927 16:28:33.050220 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.042165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.15.93.190:38643)
0927 16:28:33.042846 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.042868 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.042911 (+ 43us) server_negotiation.cc:407] Connection header received
0927 16:28:33.043617 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.043636 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.043951 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.044292 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:33.044743 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.045696 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.046567 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.047097 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.047266 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.048136 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:33.048142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:33.048157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:33.048650 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:33.048722 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:33.048751 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:33.048899 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:33.049012 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 16:28:33.049271 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260927 16:28:33.050411 20915 leader_election.cc:288] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:33.050524 20920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:33.050741 20920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [term 1 LEADER]: Becoming Leader. State: Replica: 9115993726044871884e45ac75cbf93f, State: Running, Role: LEADER
I20260927 16:28:33.051065 20920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } }
I20260927 16:28:33.051656 20915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:33.052703 20921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9115993726044871884e45ac75cbf93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } } }
I20260927 16:28:33.052816 20921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.053217 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:28:33.053237 20922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115993726044871884e45ac75cbf93f. Latest consensus state: current_term: 1 leader_uuid: "9115993726044871884e45ac75cbf93f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115993726044871884e45ac75cbf93f" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38643 } } }
I20260927 16:28:33.053313 20922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:33.056748 20935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:33.056836 20935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:33.056911 20929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:33.057538 20929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:33.062696 20929 catalog_manager.cc:1347] Generated new cluster ID: ee1c867cc92d4ad48209d4b3b4dba439
I20260927 16:28:33.062762 20929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:33.069283 20929 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:33.070123 20929 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:33.076233 20929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9115993726044871884e45ac75cbf93f: Generated new TSK 0
I20260927 16:28:33.076862 20929 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:28:33.100893 20945 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 16:28:33.101006 20945 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:28:33.101547 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.085143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.15.93.190:38643)
0927 16:28:33.085255 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.085261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.085463 (+ 202us) server_negotiation.cc:407] Connection header received
0927 16:28:33.085693 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.085696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.085748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.085848 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:28:33.086108 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.086634 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.087563 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.087794 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.087902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.088236 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:28:33.101117 (+ 12881us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:28:33.101398 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 16:28:33.101615 20918 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 16:28:33.102648 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:33.085071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38643 (local address 127.0.0.1:57348)
0927 16:28:33.085381 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:33.085393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:33.085532 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:33.085862 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:33.085865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:33.085873 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:28:33.085995 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.086001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.086758 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.086762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.087442 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.087451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.087908 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.087912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.088002 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.088049 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:28:33.101337 (+ 13288us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 16:28:33.101553 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.93.190:38643: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.93.190:38643: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 16:28:33.103590 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20848
2026-09-27T16:28:33Z chronyd exiting
I20260927 16:28:33.140548 15734 test_util.cc:175] -----------------------------------------------
I20260927 16:28:33.140645 15734 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790526488131879-15734-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 16:28:33.141548 15734 mini_oidc.cc:152] Starting JWKS server
I20260927 16:28:33.142329 15734 webserver.cc:460] Webserver started at https://127.0.0.1:37471/ using document root <none> and password file <none>
I20260927 16:28:33.142405 15734 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:28:33.142654 15734 webserver.cc:460] Webserver started at http://127.0.0.1:36715/ using document root <none> and password file <none>
2026-09-27T16:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:33Z Disabled control of system clock
I20260927 16:28:33.150792 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42645
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:42645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/testchainca.cert with env {}
W20260927 16:28:33.256716 20951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:33.257018 20951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:33.257062 20951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:33.261015 20951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:33.261066 20951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:33.261082 20951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:33.261098 20951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:33.261113 20951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:28:33.261129 20951 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:28:33.261142 20951 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37471/jwks.json
I20260927 16:28:33.266273 20951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:42645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:33.267212 20951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:33.268224 20951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:33.276300 20957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.276311 20959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.276371 20956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:33.276496 20951 server_base.cc:1034] running on GCE node
I20260927 16:28:33.277179 20951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:33.278239 20951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:33.279450 20951 hybrid_clock.cc:648] HybridClock initialized: now 1790526513279421 us; error 46 us; skew 500 ppm
I20260927 16:28:33.281255 20951 webserver.cc:460] Webserver started at http://127.15.93.190:39123/ using document root <none> and password file <none>
I20260927 16:28:33.281821 20951 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:33.281913 20951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:33.282140 20951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:33.284226 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "e7b37eb605d04787b3a536a874b66884"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "a26aca3ec0c55500044cc0e4f9d57063"
server_key_iv: "845d63f298508a650f43df6c4fd1f86a"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.284801 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "e7b37eb605d04787b3a536a874b66884"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "a26aca3ec0c55500044cc0e4f9d57063"
server_key_iv: "845d63f298508a650f43df6c4fd1f86a"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.288425 20951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:33.290783 20965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:33.292040 20951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:33.292179 20951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "e7b37eb605d04787b3a536a874b66884"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "a26aca3ec0c55500044cc0e4f9d57063"
server_key_iv: "845d63f298508a650f43df6c4fd1f86a"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.292294 20951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:33.307214 20951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:33.307966 20951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:33.308172 20951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:33.314936 20951 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:42645
I20260927 16:28:33.314945 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:42645 every 8 connection(s)
I20260927 16:28:33.315964 20951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:33.319090 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:33.324417 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: Bootstrap starting.
I20260927 16:28:33.325850 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 20951
I20260927 16:28:33.325958 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:33.326261 15734 external_mini_cluster.cc:1428] Setting key 8840e014eaef7f2a2e66eaced3ff5a49
I20260927 16:28:33.327054 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:33.327951 21018 log.cc:824] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:33.329830 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: No bootstrap required, opened a new log
I20260927 16:28:33.333244 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } }
I20260927 16:28:33.333465 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:33.333524 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b37eb605d04787b3a536a874b66884, State: Initialized, Role: FOLLOWER
I20260927 16:28:33.333988 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } }
I20260927 16:28:33.334136 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:33.334210 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:33.334329 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:33.335191 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.327310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.15.93.190:42645)
0927 16:28:33.327972 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.327986 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.328045 (+ 59us) server_negotiation.cc:407] Connection header received
0927 16:28:33.328690 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.328706 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.329027 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.329358 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:33.329777 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.330771 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.331760 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.332224 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.332393 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.333239 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:33.333244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:33.333257 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:33.333729 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:33.333787 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:33.333807 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:33.333913 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:33.334027 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 16:28:33.334467 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
I20260927 16:28:33.335256 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } }
I20260927 16:28:33.335614 21018 leader_election.cc:302] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b37eb605d04787b3a536a874b66884; no voters:
I20260927 16:28:33.335966 21018 leader_election.cc:288] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:33.336042 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:33.337035 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [term 1 LEADER]: Becoming Leader. State: Replica: e7b37eb605d04787b3a536a874b66884, State: Running, Role: LEADER
I20260927 16:28:33.337232 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:33.337400 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } }
I20260927 16:28:33.338143 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:28:33.338586 21023 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b37eb605d04787b3a536a874b66884. Latest consensus state: current_term: 1 leader_uuid: "e7b37eb605d04787b3a536a874b66884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } } }
I20260927 16:28:33.338691 21023 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.339815 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7b37eb605d04787b3a536a874b66884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b37eb605d04787b3a536a874b66884" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42645 } } }
I20260927 16:28:33.340246 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884 [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:33.342466 21038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:33.342557 21038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:33.342615 21032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:33.343483 21032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:33.348675 21032 catalog_manager.cc:1347] Generated new cluster ID: 2bdd6b2f728f46d3a2269b345abc202a
I20260927 16:28:33.348744 21032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:33.358923 21032 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:33.359892 21032 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:33.365796 21032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b37eb605d04787b3a536a874b66884: Generated new TSK 0
I20260927 16:28:33.366459 21032 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 16:28:33.402001 21048 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 16:28:33.402108 21048 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 16:28:33.402840 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.380956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.15.93.190:42645)
0927 16:28:33.381089 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.381094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.381235 (+ 141us) server_negotiation.cc:407] Connection header received
0927 16:28:33.381362 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.381366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.381420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.381538 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:28:33.381739 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.382574 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.383604 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.383861 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.383986 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.384298 (+ 312us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:28:33.402401 (+ 18103us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:28:33.402690 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 16:28:33.402915 21021 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 16:28:33.402949 21047 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:33.380842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:42645 (local address 127.0.0.1:51678)
0927 16:28:33.381151 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:33.381164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:33.381258 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:33.381538 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:33.381540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:33.381546 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:28:33.381635 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.381640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.382754 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.382758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.383466 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.383483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.383986 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.383990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.384088 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.384111 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:28:33.402605 (+ 18494us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 16:28:33.402776 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.93.190:42645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260927 16:28:33.403748 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 20951
2026-09-27T16:28:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 16:28:33.442329 15734 mini_oidc.cc:152] Starting JWKS server
I20260927 16:28:33.443137 15734 webserver.cc:460] Webserver started at https://127.0.0.1:43133/ using document root <none> and password file <none>
I20260927 16:28:33.443219 15734 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 16:28:33.443501 15734 webserver.cc:460] Webserver started at http://127.0.0.1:38299/ using document root <none> and password file <none>
2026-09-27T16:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:33Z Disabled control of system clock
I20260927 16:28:33.451814 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:46671
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:46671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43133/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:28:33.557356 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:33.557596 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:33.557633 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:33.561595 21054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:33.561643 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:33.561658 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:33.561713 21054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:33.561748 21054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:28:33.561786 21054 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:28:33.561829 21054 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43133/jwks.json
I20260927 16:28:33.566376 21054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:46671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43133/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:33.567333 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:33.568652 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:33.576699 21060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.576721 21062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.576707 21059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:33.576804 21054 server_base.cc:1034] running on GCE node
I20260927 16:28:33.577433 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:33.578455 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:33.579643 21054 hybrid_clock.cc:648] HybridClock initialized: now 1790526513579618 us; error 45 us; skew 500 ppm
I20260927 16:28:33.581290 21054 webserver.cc:460] Webserver started at http://127.15.93.190:44673/ using document root <none> and password file <none>
I20260927 16:28:33.581846 21054 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:33.581933 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:33.582156 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:33.584218 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "3049ac40220f47b19751217a50b85a5b"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "cbb1553ec5c90cc0fb1b615e41df243f"
server_key_iv: "24b1f741f916be59e9a628a833ce3f57"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.584762 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "3049ac40220f47b19751217a50b85a5b"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "cbb1553ec5c90cc0fb1b615e41df243f"
server_key_iv: "24b1f741f916be59e9a628a833ce3f57"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.588131 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:33.590276 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:33.591341 21054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:33.591504 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "3049ac40220f47b19751217a50b85a5b"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "cbb1553ec5c90cc0fb1b615e41df243f"
server_key_iv: "24b1f741f916be59e9a628a833ce3f57"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.591621 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:33.605753 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:33.606331 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:33.606513 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:33.613003 21054 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:46671
I20260927 16:28:33.613009 21120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:46671 every 8 connection(s)
I20260927 16:28:33.613847 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:33.616406 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21054
I20260927 16:28:33.616430 21121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:33.616552 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:33.616818 15734 external_mini_cluster.cc:1428] Setting key e19b7f14efe326ead1314b746bf50e15
I20260927 16:28:33.622784 21121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b: Bootstrap starting.
I20260927 16:28:33.625458 21121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:33.625502 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.617954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.15.93.190:46671)
0927 16:28:33.618502 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.618517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.618548 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:28:33.619169 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.619185 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.619604 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.619912 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:33.620306 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.621228 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.622102 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.622558 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.622727 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.623613 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:33.623627 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:33.623641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:33.624111 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:33.624180 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:33.624210 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:33.624312 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:33.624414 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 16:28:33.624667 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260927 16:28:33.626294 21121 log.cc:824] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:33.628189 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:28:33.628362 21121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b: No bootstrap required, opened a new log
I20260927 16:28:33.631770 21121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } }
I20260927 16:28:33.631987 21121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:33.632064 21121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3049ac40220f47b19751217a50b85a5b, State: Initialized, Role: FOLLOWER
I20260927 16:28:33.632462 21121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } }
I20260927 16:28:33.632601 21121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:33.632691 21121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:33.632803 21121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:33.633613 21121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } }
I20260927 16:28:33.633924 21121 leader_election.cc:302] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3049ac40220f47b19751217a50b85a5b; no voters:
I20260927 16:28:33.634240 21121 leader_election.cc:288] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:33.634335 21126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:33.634603 21126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [term 1 LEADER]: Becoming Leader. State: Replica: 3049ac40220f47b19751217a50b85a5b, State: Running, Role: LEADER
I20260927 16:28:33.634927 21126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } }
I20260927 16:28:33.635452 21121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:33.636675 21127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3049ac40220f47b19751217a50b85a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } } }
I20260927 16:28:33.636675 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3049ac40220f47b19751217a50b85a5b. Latest consensus state: current_term: 1 leader_uuid: "3049ac40220f47b19751217a50b85a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3049ac40220f47b19751217a50b85a5b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 46671 } } }
I20260927 16:28:33.636843 21127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.636843 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.637279 21137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:33.640259 21137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:33.644966 21137 catalog_manager.cc:1347] Generated new cluster ID: f77bd5bc63c64bca8d0c573a05ab700e
I20260927 16:28:33.645023 21137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:33.654348 21137 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:33.655544 21137 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:33.663545 21137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3049ac40220f47b19751217a50b85a5b: Generated new TSK 0
I20260927 16:28:33.664284 21137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:28:33.675904 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.671024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.15.93.190:46671)
0927 16:28:33.671165 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.671169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.671345 (+ 176us) server_negotiation.cc:407] Connection header received
0927 16:28:33.671505 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.671507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.671551 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.671635 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:28:33.671889 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.672452 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.673671 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.673876 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.673947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.674260 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:28:33.675573 (+ 1313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:28:33.675787 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 16:28:33.675958 21124 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 16:28:33.677799 21150 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:33.670940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:46671 (local address 127.0.0.1:43952)
0927 16:28:33.671254 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:33.671269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:33.671400 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:33.671660 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:33.671663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:33.671671 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:28:33.671786 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.671793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.672584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.672588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.673563 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.673573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.673985 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.673988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.674075 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.674102 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:28:33.675732 (+ 1630us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 16:28:33.677631 (+ 1899us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.93.190:46671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1718,"thread_start_us":105,"threads_started":1}
I20260927 16:28:33.678231 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21054
E20260927 16:28:33.689193 21151 webserver.cc:533] Webserver: sslize failed:
E20260927 16:28:33.689253 21151 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T16:28:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (275 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T16:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:33Z Disabled control of system clock
I20260927 16:28:33.725998 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41115
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41115
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/testchainca.cert with env {}
W20260927 16:28:33.833912 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:33.834159 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:33.834198 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:33.838053 21155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:33.838101 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:33.838117 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:33.838133 21155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:33.838147 21155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 16:28:33.838163 21155 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 16:28:33.838176 21155 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 16:28:33.843389 21155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:33.844337 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:33.845250 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:33.853683 21163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.853724 21160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:33.853684 21161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:33.853945 21155 server_base.cc:1034] running on GCE node
I20260927 16:28:33.854562 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:33.855609 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:33.856788 21155 hybrid_clock.cc:648] HybridClock initialized: now 1790526513856758 us; error 40 us; skew 500 ppm
I20260927 16:28:33.858637 21155 webserver.cc:460] Webserver started at http://127.15.93.190:38749/ using document root <none> and password file <none>
I20260927 16:28:33.859201 21155 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:33.859261 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:33.859521 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:33.861446 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "e9fe86da18384d3486536f28e6d7895e"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "4a068814a4915726e5ed50229cc25752"
server_key_iv: "c9e40bebab7aba2f12e644be3eaf5508"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.861996 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "e9fe86da18384d3486536f28e6d7895e"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "4a068814a4915726e5ed50229cc25752"
server_key_iv: "c9e40bebab7aba2f12e644be3eaf5508"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.865500 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:33.867748 21169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:33.868942 21155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:33.869056 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "e9fe86da18384d3486536f28e6d7895e"
format_stamp: "Formatted at 2026-09-27 16:28:33 on dist-test-slave-grzf"
server_key: "4a068814a4915726e5ed50229cc25752"
server_key_iv: "c9e40bebab7aba2f12e644be3eaf5508"
server_key_version: "encryptionkey@0"
I20260927 16:28:33.869163 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:33.881297 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:33.881942 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:33.882133 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:33.888717 21155 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:41115
I20260927 16:28:33.888741 21221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:41115 every 8 connection(s)
I20260927 16:28:33.889624 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:33.891160 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21155
I20260927 16:28:33.891281 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:33.891599 15734 external_mini_cluster.cc:1428] Setting key 602ca23e8ebb7d0ccfc77a08b6e87d78
I20260927 16:28:33.892817 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:33.898890 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e: Bootstrap starting.
I20260927 16:28:33.900861 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.892840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.15.93.190:41115)
0927 16:28:33.893465 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.893480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.893513 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:28:33.894116 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.894139 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.894514 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.894784 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:33.895132 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.896125 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.897319 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.897973 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.898109 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.898965 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:33.898970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:33.898981 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:28:33.899559 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:33.899650 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:28:33.899671 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:33.899745 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:33.899901 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 16:28:33.900176 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260927 16:28:33.901322 21222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:33.902107 21222 log.cc:824] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:33.903723 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:28:33.904210 21222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e: No bootstrap required, opened a new log
I20260927 16:28:33.907872 21222 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } }
I20260927 16:28:33.908078 21222 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:33.908154 21222 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fe86da18384d3486536f28e6d7895e, State: Initialized, Role: FOLLOWER
I20260927 16:28:33.908584 21222 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } }
I20260927 16:28:33.908723 21222 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:33.908804 21222 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:33.908915 21222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:33.909766 21222 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } }
I20260927 16:28:33.910079 21222 leader_election.cc:302] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9fe86da18384d3486536f28e6d7895e; no voters:
I20260927 16:28:33.910395 21222 leader_election.cc:288] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:33.910482 21227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:33.910698 21227 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [term 1 LEADER]: Becoming Leader. State: Replica: e9fe86da18384d3486536f28e6d7895e, State: Running, Role: LEADER
I20260927 16:28:33.911020 21227 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } }
I20260927 16:28:33.911614 21222 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:33.912724 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9fe86da18384d3486536f28e6d7895e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } } }
I20260927 16:28:33.912879 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.912722 21229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9fe86da18384d3486536f28e6d7895e. Latest consensus state: current_term: 1 leader_uuid: "e9fe86da18384d3486536f28e6d7895e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9fe86da18384d3486536f28e6d7895e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41115 } } }
I20260927 16:28:33.913131 21229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:33.913246 21236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:33.915776 21236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:33.921259 21236 catalog_manager.cc:1347] Generated new cluster ID: 963d537f755b4dfb978da86ac91cb34c
I20260927 16:28:33.921329 21236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:33.936885 21236 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:33.937736 21236 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:33.946274 21236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9fe86da18384d3486536f28e6d7895e: Generated new TSK 0
I20260927 16:28:33.946924 21236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:28:34.054955 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:33.957351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.15.93.190:41115)
0927 16:28:33.957464 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:33.957469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:33.957593 (+ 124us) server_negotiation.cc:407] Connection header received
0927 16:28:33.957731 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:33.957734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:33.957787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:33.957888 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0927 16:28:33.958091 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.958897 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.959780 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.960128 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.960230 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.960473 (+ 243us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 16:28:34.054587 (+ 94114us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:28:34.054838 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 16:28:34.055039 21225 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 16:28:34.055318 21251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:33.957242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41115 (local address 127.0.0.1:54942)
0927 16:28:33.957506 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:33.957518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:33.957607 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:33.957900 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:33.957902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:33.957907 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0927 16:28:33.957997 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.958001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.958993 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.958995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.959668 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:33.959677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:33.960235 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:33.960238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:33.960296 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:33.960315 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 16:28:34.054806 (+ 94491us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 16:28:34.054983 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.93.190:41115: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260927 16:28:34.055979 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21155
2026-09-27T16:28:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (356 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:34 dist-test-slave-grzf krb5kdc[21263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:34 dist-test-slave-grzf krb5kdc[21263](info): set up 2 sockets
Sep 27 16:28:34 dist-test-slave-grzf krb5kdc[21263](info): commencing operation
krb5kdc: starting...
W20260927 16:28:35.327883 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:35 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:35.440022 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43481
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:35.546763 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:35.547190 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:35.547312 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:35.553874 21276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:35.553956 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:35.553995 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:35.554028 21276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:35.554069 21276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:35.562924 21276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:35.564352 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:35.565673 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:35.576416 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:35.576409 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:35.576433 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:35.577440 21276 server_base.cc:1034] running on GCE node
I20260927 16:28:35.578012 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:35.579070 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:35.580256 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790526515580229 us; error 47 us; skew 500 ppm
Sep 27 16:28:35 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:35.582911 21276 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:35.584010 21276 webserver.cc:460] Webserver started at http://127.15.93.190:33167/ using document root <none> and password file <none>
I20260927 16:28:35.584544 21276 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:35.584630 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:35.584853 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:35.587507 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "a99e517c02c849d0a32c6e5f7d819055"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "a0d544a0347bd267eb347ef77018702e"
server_key_iv: "202a6cbd006c48e160c63239e57391eb"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.587988 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "a99e517c02c849d0a32c6e5f7d819055"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "a0d544a0347bd267eb347ef77018702e"
server_key_iv: "202a6cbd006c48e160c63239e57391eb"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.591431 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:35.593624 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:35.594779 21276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:35.594870 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "a99e517c02c849d0a32c6e5f7d819055"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "a0d544a0347bd267eb347ef77018702e"
server_key_iv: "202a6cbd006c48e160c63239e57391eb"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.594946 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:35.611142 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:35.613829 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:35.613971 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:35.620870 21276 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:43481
I20260927 16:28:35.620895 21343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:43481 every 8 connection(s)
I20260927 16:28:35.621742 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:35.624217 21344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:35.625871 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21276
I20260927 16:28:35.626005 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:35.626269 15734 external_mini_cluster.cc:1428] Setting key 8aff6e8a1e51f84dc11e54dd5a325a04
I20260927 16:28:35.630704 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055: Bootstrap starting.
I20260927 16:28:35.633105 21344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:35.633972 21344 log.cc:824] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:28:35 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:35.635908 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055: No bootstrap required, opened a new log
I20260927 16:28:35.640409 21344 raft_consensus.cc:348] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } }
I20260927 16:28:35.640661 21344 raft_consensus.cc:374] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:35.640717 21344 raft_consensus.cc:729] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99e517c02c849d0a32c6e5f7d819055, State: Initialized, Role: FOLLOWER
I20260927 16:28:35.640790 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:35.627608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.15.93.190:43481)
0927 16:28:35.628005 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:35.628026 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:35.628066 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:28:35.628887 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:35.628910 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:35.629314 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:35.629666 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:35.630282 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.631722 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.632561 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.633236 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.633402 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:35.635986 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:35.635992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:35.636007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:35.636035 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:35.637904 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:35.638302 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:35.638308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:35.638314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:35.638442 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:35.638698 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:35.638701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:35.638703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:35.639182 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:35.639407 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:35.639603 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 16:28:35.639885 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260927 16:28:35.641166 21344 consensus_queue.cc:260] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } }
I20260927 16:28:35.641321 21344 raft_consensus.cc:388] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:35.641376 21344 raft_consensus.cc:482] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:35.641453 21344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:35.642323 21344 raft_consensus.cc:504] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } }
I20260927 16:28:35.642654 21344 leader_election.cc:302] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a99e517c02c849d0a32c6e5f7d819055; no voters:
I20260927 16:28:35.642995 21344 leader_election.cc:288] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:35.643103 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:35.643339 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [term 1 LEADER]: Becoming Leader. State: Replica: a99e517c02c849d0a32c6e5f7d819055, State: Running, Role: LEADER
I20260927 16:28:35.643654 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } }
I20260927 16:28:35.644219 21344 sys_catalog.cc:564] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:35.645275 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a99e517c02c849d0a32c6e5f7d819055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } } }
I20260927 16:28:35.645292 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a99e517c02c849d0a32c6e5f7d819055. Latest consensus state: current_term: 1 leader_uuid: "a99e517c02c849d0a32c6e5f7d819055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a99e517c02c849d0a32c6e5f7d819055" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43481 } } }
I20260927 16:28:35.645403 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:35.645406 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:35.645771 21358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:35.648813 21358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:35.653671 21358 catalog_manager.cc:1347] Generated new cluster ID: e7935a1147eb4e7f8c9dadd101e906bc
I20260927 16:28:35.653733 21358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:35.661681 21358 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:35.662520 21358 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:35.672856 21358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a99e517c02c849d0a32c6e5f7d819055: Generated new TSK 0
I20260927 16:28:35.673547 21358 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:35.699476 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:35.809666 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:35.809895 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:35.809937 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:35.814159 21370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:35.814211 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:35.814267 21370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:35.819304 21370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:35.820434 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:35.821444 21370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:35.831010 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:35.831154 21370 server_base.cc:1034] running on GCE node
W20260927 16:28:35.830971 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:35.831041 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:35.831755 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:35.832401 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:35.833557 21370 hybrid_clock.cc:648] HybridClock initialized: now 1790526515833546 us; error 29 us; skew 500 ppm
Sep 27 16:28:35 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:35.836234 21370 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:35.837076 21370 webserver.cc:460] Webserver started at http://127.15.93.129:40805/ using document root <none> and password file <none>
I20260927 16:28:35.837605 21370 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:35.837670 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:35.837900 21370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:35.839913 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "ddca5375a39a414db2158e64997d0e1c"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "c1d08ee385ddb03369abdbe84493c2bc"
server_key_iv: "f6590be86823ea3d2e0c661723e406d0"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.840457 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "ddca5375a39a414db2158e64997d0e1c"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "c1d08ee385ddb03369abdbe84493c2bc"
server_key_iv: "f6590be86823ea3d2e0c661723e406d0"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.844203 21370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:35.846722 21385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:35.847920 21370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:35.848037 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "ddca5375a39a414db2158e64997d0e1c"
format_stamp: "Formatted at 2026-09-27 16:28:35 on dist-test-slave-grzf"
server_key: "c1d08ee385ddb03369abdbe84493c2bc"
server_key_iv: "f6590be86823ea3d2e0c661723e406d0"
server_key_version: "encryptionkey@0"
I20260927 16:28:35.848140 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:35.881739 21370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:35.884963 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:35.885160 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:35.885746 21370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:35.886552 21370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:35.886598 21370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:35.886669 21370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:35.886703 21370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:35.895439 21370 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:36297
I20260927 16:28:35.895516 21498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:36297 every 8 connection(s)
I20260927 16:28:35.896346 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:35.905453 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21370
I20260927 16:28:35.905606 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
Sep 27 16:28:35 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:35.905903 15734 external_mini_cluster.cc:1428] Setting key ebfaa4c9aff79a194381f1c26eb9e896
I20260927 16:28:35.910445 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:35.898103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:33815 (local address 127.15.93.190:43481)
0927 16:28:35.898234 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:35.898238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:35.899167 (+ 929us) server_negotiation.cc:407] Connection header received
0927 16:28:35.900529 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:35.900533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:35.900589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:35.900702 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:35.901865 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.902399 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.903563 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.903804 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.903916 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:35.906520 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:35.906524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:35.906528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:35.906554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:35.908292 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:35.908783 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:35.908790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:35.908794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:35.908885 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:35.909321 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:35.909323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:35.909325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:35.909494 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:35.909569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:35.910226 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 16:28:35.910327 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:28:35.911712 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:35.898508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43481 (local address 127.15.93.129:33815)
0927 16:28:35.899014 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:35.899067 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:35.900323 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:35.900875 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:35.900883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:35.901238 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:35.901710 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:35.901724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.902525 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.902529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:35.903437 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:35.903446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:35.903957 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:35.903961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:35.904130 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:35.904674 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:35.904690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:35.906359 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:35.908419 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:35.908423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:35.908436 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:35.908688 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:35.908991 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:35.908996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:35.909000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:35.909215 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:35.909604 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:35.909613 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:35.909950 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 16:28:35.910196 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 16:28:35.910557 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":100,"threads_started":1}
I20260927 16:28:35.913004 21499 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43481
I20260927 16:28:35.913241 21499 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:35.913727 21499 heartbeater.cc:511] Master 127.15.93.190:43481 requested a full tablet report, sending...
I20260927 16:28:35.915282 21308 ts_manager.cc:194] Registered new tserver with Master: ddca5375a39a414db2158e64997d0e1c (127.15.93.129:36297)
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:35.916442 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:33815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:35.932021 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:36.039721 21504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:36.039988 21504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:36.040074 21504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:36.044080 21504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:36.044155 21504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:36.044262 21504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:36.049578 21504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:36.050554 21504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:36.051612 21504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:36.060693 21509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:36.060758 21510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:36.060693 21512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:36.060868 21504 server_base.cc:1034] running on GCE node
I20260927 16:28:36.061439 21504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:36.062005 21504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:36.063189 21504 hybrid_clock.cc:648] HybridClock initialized: now 1790526516063181 us; error 31 us; skew 500 ppm
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.065763 21504 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:36.066525 21504 webserver.cc:460] Webserver started at http://127.15.93.130:37359/ using document root <none> and password file <none>
I20260927 16:28:36.067099 21504 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:36.067188 21504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:36.067463 21504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:36.069360 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "1982920fb40d4633946c0b5125ca1b3a"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "68c31e077d50c126a0561e2ccfa28c2a"
server_key_iv: "1a7cf84044cd1f60c5850ee72467b83a"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.069955 21504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "1982920fb40d4633946c0b5125ca1b3a"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "68c31e077d50c126a0561e2ccfa28c2a"
server_key_iv: "1a7cf84044cd1f60c5850ee72467b83a"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.073477 21504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:36.075733 21519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.076857 21504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:36.076984 21504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "1982920fb40d4633946c0b5125ca1b3a"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "68c31e077d50c126a0561e2ccfa28c2a"
server_key_iv: "1a7cf84044cd1f60c5850ee72467b83a"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.077091 21504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:36.108719 21504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:36.111606 21504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:36.111800 21504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:36.112373 21504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:36.113157 21504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:36.113226 21504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.113291 21504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:36.113341 21504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.121788 21504 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:39811
I20260927 16:28:36.121814 21632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:39811 every 8 connection(s)
I20260927 16:28:36.122622 21504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:36.127954 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21504
I20260927 16:28:36.128079 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:36.128386 15734 external_mini_cluster.cc:1428] Setting key 42e9342d577aeb0c8a7c3406e588a600
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:36.136603 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.124452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41453 (local address 127.15.93.190:43481)
0927 16:28:36.124584 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.124588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.125359 (+ 771us) server_negotiation.cc:407] Connection header received
0927 16:28:36.126586 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.126589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.126647 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.126719 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.127795 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.128571 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.129891 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.130253 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.130366 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.133026 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:36.133029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:36.133031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:36.133052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:36.134448 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.135080 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.135082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.135084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.135153 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.135570 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.135572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.135573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.135743 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:36.135818 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:36.136401 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 16:28:36.136492 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:28:36.137706 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.124814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43481 (local address 127.15.93.130:41453)
0927 16:28:36.125219 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:36.125259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:36.126408 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:36.126873 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:36.126880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:36.127210 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:36.127666 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:36.127679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.128721 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.128728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:36.129780 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:36.129790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.130393 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.130395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:36.130525 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.131066 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:36.131081 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:36.132876 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:36.134575 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:36.134581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:36.134597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:36.134980 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:36.135250 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:36.135256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:36.135261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:36.135471 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:36.135848 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:36.135854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:36.136149 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 16:28:36.136356 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 16:28:36.136687 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:36.138779 21633 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43481
I20260927 16:28:36.138998 21633 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:36.139528 21633 heartbeater.cc:511] Master 127.15.93.190:43481 requested a full tablet report, sending...
I20260927 16:28:36.140703 21308 ts_manager.cc:194] Registered new tserver with Master: 1982920fb40d4633946c0b5125ca1b3a (127.15.93.130:39811)
I20260927 16:28:36.141553 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:41453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:36.158898 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:36.265270 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:36.265537 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:36.265620 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:36.269587 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:36.269660 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:36.269742 21638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:36.274729 21638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:36.275730 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:36.276669 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:36.285858 21643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:36.285872 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:36.285872 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:36.286114 21638 server_base.cc:1034] running on GCE node
I20260927 16:28:36.286574 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:36.287101 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:36.288287 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790526516288256 us; error 49 us; skew 500 ppm
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.290980 21638 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:36.291819 21638 webserver.cc:460] Webserver started at http://127.15.93.131:46825/ using document root <none> and password file <none>
I20260927 16:28:36.292346 21638 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:36.292430 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:36.292654 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:36.294566 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "4a038c0236b44ff798e47fd4942e81d3"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "c4f430f5f5779044845605ba0b1be3ed"
server_key_iv: "906c878f6d4489f806465ed23803a272"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.295118 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "4a038c0236b44ff798e47fd4942e81d3"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "c4f430f5f5779044845605ba0b1be3ed"
server_key_iv: "906c878f6d4489f806465ed23803a272"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.298645 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:36.300961 21653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.302083 21638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:36.302217 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "4a038c0236b44ff798e47fd4942e81d3"
format_stamp: "Formatted at 2026-09-27 16:28:36 on dist-test-slave-grzf"
server_key: "c4f430f5f5779044845605ba0b1be3ed"
server_key_iv: "906c878f6d4489f806465ed23803a272"
server_key_version: "encryptionkey@0"
I20260927 16:28:36.302320 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:36.322758 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:36.325827 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:36.326021 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:36.326555 21638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:36.327313 21638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:36.327410 21638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.327497 21638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:36.327550 21638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:36.335754 21638 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:39447
I20260927 16:28:36.335774 21766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:39447 every 8 connection(s)
I20260927 16:28:36.336583 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:36.344805 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21638
I20260927 16:28:36.344959 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:36.345256 15734 external_mini_cluster.cc:1428] Setting key eede1adfdf5dba6eae7c2f902131c9c7
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:36.349206 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.338239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:54029 (local address 127.15.93.190:43481)
0927 16:28:36.338365 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.338368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.339097 (+ 729us) server_negotiation.cc:407] Connection header received
0927 16:28:36.340372 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.340375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.340422 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.340496 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.341550 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.342059 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.343133 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.343336 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.343447 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.345953 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:36.345956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:36.345959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:36.345985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:36.347625 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.348045 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.348047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.348048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.348116 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.348361 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.348363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.348364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.348518 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:36.348589 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:36.348997 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 16:28:36.349083 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:28:36.349951 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.338535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43481 (local address 127.15.93.131:54029)
0927 16:28:36.338943 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:36.338982 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:36.340187 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:36.340659 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:36.340665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:36.340990 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:36.341431 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:36.341444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.342160 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.342162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:36.343026 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:36.343034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.343514 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.343520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:36.343750 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.344224 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:36.344239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:36.345796 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:36.347718 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:36.347722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:36.347732 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:36.347965 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:36.348179 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:36.348181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:36.348183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:36.348291 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:36.348602 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:36.348606 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:36.348813 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 16:28:36.348954 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 16:28:36.349173 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 16:28:36.350991 21767 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43481
I20260927 16:28:36.351272 21767 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:36.351800 21767 heartbeater.cc:511] Master 127.15.93.190:43481 requested a full tablet report, sending...
I20260927 16:28:36.352826 21308 ts_manager.cc:194] Registered new tserver with Master: 4a038c0236b44ff798e47fd4942e81d3 (127.15.93.131:39447)
I20260927 16:28:36.353392 21308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:54029
I20260927 16:28:36.358999 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/test-admin.keytab.
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.375620 15734 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 16:28:36.378166 21774 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:36.376831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.15.93.129:36297)
0927 16:28:36.377127 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.377141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.377163 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:28:36.377209 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.377213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.377376 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.377488 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.377963 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47834: BlockingRecv error: recv got EOF from 127.0.0.1:47834 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
W20260927 16:28:36.378190 15734 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.379284 15734 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.380255 15734 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:36.389206 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.381004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.15.93.129:36297)
0927 16:28:36.381130 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.381134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.381252 (+ 118us) server_negotiation.cc:407] Connection header received
0927 16:28:36.381322 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.381324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.381359 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.381416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.381846 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.382590 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.383338 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.383727 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.383808 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.385876 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:36.385885 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:36.385896 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:36.385916 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:36.387688 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.388066 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.388071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.388073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.388161 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.388377 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.388379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.388380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.388590 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:36.388732 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:36.388880 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 16:28:36.389046 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:28:36.392853 21774 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:36.391881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.15.93.129:36297)
0927 16:28:36.391998 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.392002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.392094 (+ 92us) server_negotiation.cc:407] Connection header received
0927 16:28:36.392239 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.392244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.392283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.392363 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.392748 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47856: BlockingRecv error: recv got EOF from 127.0.0.1:47856 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 16:28:36.392980 15734 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.393913 15734 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.394757 15734 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:36.402498 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.395472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.15.93.129:36297)
0927 16:28:36.395610 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.395613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.395678 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:28:36.395754 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.395756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.395786 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.395842 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.396249 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.396740 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.397470 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.397686 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.397760 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.399974 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:36.399976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:36.399978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:36.399999 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:36.401174 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.401512 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.401514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.401515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.401580 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.401797 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.401799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.401800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.401945 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:36.402038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:36.402222 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:28:36.402335 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 16:28:36.404958 21774 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:36.404058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.15.93.129:36297)
0927 16:28:36.404182 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.404186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.404255 (+ 69us) server_negotiation.cc:407] Connection header received
0927 16:28:36.404364 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.404367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.404401 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.404490 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.404856 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47880: BlockingRecv error: recv got EOF from 127.0.0.1:47880 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 16:28:36.405107 15734 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790526488131879-15734-0/krb5kdc/krb5cc' of type FILE
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.406095 15734 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:36.406970 15734 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:28:36.414601 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:36.407717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.15.93.129:36297)
0927 16:28:36.407839 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:36.407842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:36.407886 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:28:36.407965 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:36.407967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:36.407997 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:36.408052 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:36.408445 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.408899 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.409582 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:36.409778 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:36.409848 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:36.411858 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:36.411861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:36.411863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:36.411882 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:36.413052 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.413485 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.413489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.413491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.413567 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:36.413835 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:36.413839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:36.413841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:36.413992 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:36.414118 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:36.414349 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 16:28:36.414471 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:36.415711 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21370
I20260927 16:28:36.421991 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21504
I20260927 16:28:36.427348 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21638
I20260927 16:28:36.432528 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21276
2026-09-27T16:28:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2138 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21787](info): set up 2 sockets
Sep 27 16:28:36 dist-test-slave-grzf krb5kdc[21787](info): commencing operation
krb5kdc: starting...
W20260927 16:28:37.467929 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:37 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab.
Entry for principal oryx/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab.
I20260927 16:28:37.583309 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44091
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab
--principal=oryx/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:37.688760 21800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:37.689002 21800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:37.689039 21800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:37.692927 21800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:37.692976 21800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:37.692996 21800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:37.693011 21800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:37.693027 21800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:37.698197 21800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab
--principal=oryx/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:37.699134 21800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:37.700107 21800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:37.708087 21805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:37.708143 21806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:37.708087 21808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:37.708247 21800 server_base.cc:1034] running on GCE node
I20260927 16:28:37.708899 21800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:37.709906 21800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:37.711046 21800 hybrid_clock.cc:648] HybridClock initialized: now 1790526517711035 us; error 23 us; skew 500 ppm
Sep 27 16:28:37 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:37.713680 21800 init.cc:377] Logged in from keytab as oryx/127.15.93.190@KRBTEST.COM (short username oryx)
I20260927 16:28:37.714548 21800 webserver.cc:460] Webserver started at http://127.15.93.190:41621/ using document root <none> and password file <none>
I20260927 16:28:37.715081 21800 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:37.715173 21800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:37.715420 21800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:37.717348 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "be0ed8ce3c0c44c0912336d08e3acc72"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "8488f2a68ac73a95f21ed5388ca013a6"
server_key_iv: "70f6fb782daeb53a6bf8e4ebc97ba7b2"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.717912 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "be0ed8ce3c0c44c0912336d08e3acc72"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "8488f2a68ac73a95f21ed5388ca013a6"
server_key_iv: "70f6fb782daeb53a6bf8e4ebc97ba7b2"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.721366 21800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:37.723618 21815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:37.724777 21800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:37.724910 21800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "be0ed8ce3c0c44c0912336d08e3acc72"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "8488f2a68ac73a95f21ed5388ca013a6"
server_key_iv: "70f6fb782daeb53a6bf8e4ebc97ba7b2"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.725013 21800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:37.747639 21800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:37.750504 21800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:37.750655 21800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:37.757438 21800 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:44091
I20260927 16:28:37.757706 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:44091 every 8 connection(s)
I20260927 16:28:37.758421 21800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:37.758741 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21800
I20260927 16:28:37.758865 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:37.759177 15734 external_mini_cluster.cc:1428] Setting key aea2d88ca0ed10bfd834ff12a68a398c
I20260927 16:28:37.761390 21868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:37.767781 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Bootstrap starting.
Sep 27 16:28:37 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:37.770092 21868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:37.770789 21868 log.cc:824] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:37.772734 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: No bootstrap required, opened a new log
I20260927 16:28:37.773134 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:37.760600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.15.93.190:44091)
0927 16:28:37.760974 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:37.760989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:37.761021 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:28:37.761692 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:37.761709 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:37.762017 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:37.762283 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:37.763121 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:37.764168 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:37.765318 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:37.765761 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:37.765867 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:37.768566 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:37.768570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:37.768581 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:37.768605 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:37.770233 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:37.770672 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:37.770684 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:37.770689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:37.770791 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:37.771070 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:37.771073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:37.771075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:37.771664 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:37.771875 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:37.772135 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 16:28:37.772407 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260927 16:28:37.776194 21868 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } }
I20260927 16:28:37.776422 21868 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:37.776499 21868 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0ed8ce3c0c44c0912336d08e3acc72, State: Initialized, Role: FOLLOWER
I20260927 16:28:37.776937 21868 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } }
I20260927 16:28:37.777081 21868 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:37.777168 21868 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:37.777261 21868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:37.778083 21868 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } }
I20260927 16:28:37.778411 21868 leader_election.cc:302] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0ed8ce3c0c44c0912336d08e3acc72; no voters:
I20260927 16:28:37.778703 21868 leader_election.cc:288] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:37.778820 21873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:37.779057 21873 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 LEADER]: Becoming Leader. State: Replica: be0ed8ce3c0c44c0912336d08e3acc72, State: Running, Role: LEADER
I20260927 16:28:37.779428 21873 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } }
I20260927 16:28:37.779978 21868 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:37.781041 21875 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0ed8ce3c0c44c0912336d08e3acc72. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } }
I20260927 16:28:37.781193 21875 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:37.781523 21882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:37.781543 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } }
I20260927 16:28:37.782198 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:37.784538 21882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:37.790758 21882 catalog_manager.cc:1347] Generated new cluster ID: 36bf8cf5459a4de38a75de51df30a930
I20260927 16:28:37.790828 21882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:37.802610 21882 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:37.803841 21882 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:37.815327 21882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Generated new TSK 0
I20260927 16:28:37.816123 21882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
I20260927 16:28:37.842708 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:37.949524 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:37.949779 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:37.949847 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:37.953881 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:37.953953 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:37.954048 21894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:37.959136 21894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:37.960173 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:37.961149 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:37.970213 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:37.970263 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:37.970405 21894 server_base.cc:1034] running on GCE node
W20260927 16:28:37.970265 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:37.970835 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:37.971478 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:37.972657 21894 hybrid_clock.cc:648] HybridClock initialized: now 1790526517972641 us; error 29 us; skew 500 ppm
Sep 27 16:28:37 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:37.975256 21894 init.cc:377] Logged in from keytab as oryx/127.15.93.129@KRBTEST.COM (short username oryx)
I20260927 16:28:37.976079 21894 webserver.cc:460] Webserver started at http://127.15.93.129:33909/ using document root <none> and password file <none>
I20260927 16:28:37.976569 21894 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:37.976631 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:37.976785 21894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:37.978658 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "b09bcd3a45014e7998b2d0ca225ffa69"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "9bc74b8364999e3c703d7f0a5bbb3c36"
server_key_iv: "0c16c1514bc2c18fbdecd12b0e93f7ab"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.979139 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "b09bcd3a45014e7998b2d0ca225ffa69"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "9bc74b8364999e3c703d7f0a5bbb3c36"
server_key_iv: "0c16c1514bc2c18fbdecd12b0e93f7ab"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.982865 21894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 16:28:37.985069 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:37.986224 21894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:37.986346 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "b09bcd3a45014e7998b2d0ca225ffa69"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "9bc74b8364999e3c703d7f0a5bbb3c36"
server_key_iv: "0c16c1514bc2c18fbdecd12b0e93f7ab"
server_key_version: "encryptionkey@0"
I20260927 16:28:37.986440 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.000648 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.003585 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:38.003783 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.004354 21894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:38.005144 21894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:38.005190 21894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.005255 21894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:38.005286 21894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.013705 21894 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:45823
I20260927 16:28:38.013737 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:45823 every 8 connection(s)
I20260927 16:28:38.014544 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:38.018643 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 21894
I20260927 16:28:38.018786 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:38.019090 15734 external_mini_cluster.cc:1428] Setting key b1ed61a94eb3b4165a1755207191161c
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:28:38.028801 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.016215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:40961 (local address 127.15.93.190:44091)
0927 16:28:38.016352 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.016355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.017125 (+ 770us) server_negotiation.cc:407] Connection header received
0927 16:28:38.018371 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.018374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.018418 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.018481 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.019899 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.020437 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.021559 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.021958 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.022077 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.024868 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.024871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.024873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.024896 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.026435 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.027047 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.027051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.027052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.027140 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.027565 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.027568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.027570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.027778 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.027869 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.028550 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.028662 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:28:38.029634 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.016561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:40961)
0927 16:28:38.016976 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.017018 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.018186 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.018670 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.018679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.019177 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.019754 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.019767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.020564 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.020569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.021428 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.021436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.022108 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.022111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.022266 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.022856 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.022871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.024686 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.026554 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.026559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.026573 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.026932 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.027247 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.027250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.027253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.027463 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.027891 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.027898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.028216 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.028465 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.028818 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260927 16:28:38.030742 22023 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:38.030970 22023 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:38.031484 22023 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.130@KRBTEST.COM" created.
I20260927 16:28:38.033075 21832 ts_manager.cc:194] Registered new tserver with Master: b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823)
I20260927 16:28:38.034162 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:40961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
I20260927 16:28:38.045698 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:38.155330 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:38.155699 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:38.155790 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:38.160710 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:38.160772 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:38.160866 22028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:38.166092 22028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:38.167098 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:38.168140 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:28:38.177413 22028 server_base.cc:1034] running on GCE node
W20260927 16:28:38.177366 22033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.177377 22034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.177366 22036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.178048 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:38.178663 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:38.179841 22028 hybrid_clock.cc:648] HybridClock initialized: now 1790526518179826 us; error 33 us; skew 500 ppm
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:38.182574 22028 init.cc:377] Logged in from keytab as oryx/127.15.93.130@KRBTEST.COM (short username oryx)
I20260927 16:28:38.183346 22028 webserver.cc:460] Webserver started at http://127.15.93.130:44897/ using document root <none> and password file <none>
I20260927 16:28:38.183898 22028 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.183954 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:38.184104 22028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:38.185994 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "72df7b22a1d94e02b9dca37e93fc30ab"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "7f5ea379cbf20909d9153dc16ed9cd7c"
server_key_iv: "bb140a40c1153bafa728eda44836c183"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.186474 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "72df7b22a1d94e02b9dca37e93fc30ab"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "7f5ea379cbf20909d9153dc16ed9cd7c"
server_key_iv: "bb140a40c1153bafa728eda44836c183"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.190018 22028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:28:38.192480 22043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.193656 22028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:38.193783 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "72df7b22a1d94e02b9dca37e93fc30ab"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "7f5ea379cbf20909d9153dc16ed9cd7c"
server_key_iv: "bb140a40c1153bafa728eda44836c183"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.193886 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.208370 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.211179 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:38.211362 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.211961 22028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:38.212729 22028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:38.212800 22028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.212864 22028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:38.212911 22028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.221487 22028 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:43297
I20260927 16:28:38.221513 22156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:43297 every 8 connection(s)
I20260927 16:28:38.222326 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:38.231294 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22028
I20260927 16:28:38.231443 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:38.231765 15734 external_mini_cluster.cc:1428] Setting key 55748953e1d82323f33f17eb44f3e756
I20260927 16:28:38.235809 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.223990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:51953 (local address 127.15.93.190:44091)
0927 16:28:38.224111 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.224114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.224893 (+ 779us) server_negotiation.cc:407] Connection header received
0927 16:28:38.226216 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.226220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.226273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.226351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.227481 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.228093 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.229227 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.229471 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.229557 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.231913 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.231916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.231919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.231944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.233552 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.234149 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.234151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.234153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.234224 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.234605 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.234607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.234607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.234848 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.234956 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.235454 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.235592 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:28:38.236573 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.224340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:51953)
0927 16:28:38.224754 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.224795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.226009 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.226510 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.226516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.226842 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.227293 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.227307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.228202 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.228205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.229042 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.229051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.229600 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.229602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.229734 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.230210 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.230224 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.231747 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.233669 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.233677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.233693 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.234047 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.234312 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.234317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.234323 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.234516 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.234957 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.234962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.235177 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.235327 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.235632 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260927 16:28:38.237716 22157 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:38.237957 22157 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:38.238409 22157 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
I20260927 16:28:38.239408 21832 ts_manager.cc:194] Registered new tserver with Master: 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:38.239972 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:51953
WARNING: no policy specified for oryx/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
I20260927 16:28:38.257836 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:38.367182 22162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:38.367448 22162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:38.367529 22162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:38.371495 22162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:38.371548 22162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:38.371659 22162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:38.377046 22162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:38.378019 22162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:38.378996 22162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:38.388257 22168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.388278 22170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.388278 22167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.388469 22162 server_base.cc:1034] running on GCE node
I20260927 16:28:38.388895 22162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:38.389461 22162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:38.390679 22162 hybrid_clock.cc:648] HybridClock initialized: now 1790526518390659 us; error 42 us; skew 500 ppm
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:38.393323 22162 init.cc:377] Logged in from keytab as oryx/127.15.93.131@KRBTEST.COM (short username oryx)
I20260927 16:28:38.394104 22162 webserver.cc:460] Webserver started at http://127.15.93.131:34429/ using document root <none> and password file <none>
I20260927 16:28:38.394678 22162 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.394768 22162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:38.394994 22162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:38.397025 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "18df5caabaed4b89903d368d5f8bdee3"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ee199b4fa4fab32346fe8e0b32d93a38"
server_key_iv: "bc76d8d34aa6f5b4dccdf3780fee8bee"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.397591 22162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "18df5caabaed4b89903d368d5f8bdee3"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ee199b4fa4fab32346fe8e0b32d93a38"
server_key_iv: "bc76d8d34aa6f5b4dccdf3780fee8bee"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.401074 22162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:38.403450 22177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.404597 22162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:38.404732 22162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "18df5caabaed4b89903d368d5f8bdee3"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ee199b4fa4fab32346fe8e0b32d93a38"
server_key_iv: "bc76d8d34aa6f5b4dccdf3780fee8bee"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.404835 22162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.429798 22162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.432961 22162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:38.433106 22162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.433643 22162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:38.434391 22162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:38.434434 22162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.434485 22162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:38.434506 22162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.442783 22162 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:44169
I20260927 16:28:38.442845 22290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:44169 every 8 connection(s)
I20260927 16:28:38.443756 22162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:38.443943 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22162
I20260927 16:28:38.444053 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:38.444424 15734 external_mini_cluster.cc:1428] Setting key c433b1658ed099096cd4a42118f31012
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:38.457743 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.445802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38723 (local address 127.15.93.190:44091)
0927 16:28:38.445937 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.445941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.446956 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 16:28:38.448354 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.448357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.448410 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.448488 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.449605 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.450135 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.451260 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.451574 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.451685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.454112 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.454116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.454119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.454150 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.455605 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.456121 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.456127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.456129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.456211 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.456551 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.456556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.456557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.456746 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.456879 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.457454 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.457577 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:38.458645 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.446287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:38723)
0927 16:28:38.446824 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.446865 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.448155 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.448652 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.448659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.449008 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.449463 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.449475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.450262 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.450266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.451130 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.451140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.451724 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.451729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.451898 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.452444 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.452459 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.453941 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.455722 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.455728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.455742 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.456001 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.456311 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.456314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.456316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.456454 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.456895 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.456900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.457209 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.457411 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.457743 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260927 16:28:38.459758 22291 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:38.460011 22291 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:38.460490 22291 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
I20260927 16:28:38.461436 21832 ts_manager.cc:194] Registered new tserver with Master: 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:38.461982 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:38723
I20260927 16:28:38.469919 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.93.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.189.keytab.
Entry for principal oryx/127.15.93.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.189.keytab.
I20260927 16:28:38.494573 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:37363
--webserver_interface=127.15.93.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.189.keytab
--principal=oryx/127.15.93.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44091,127.15.93.189:37363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:38.600152 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:38.600409 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:38.600485 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:38.604344 22296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:38.604420 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:38.604465 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:38.604525 22296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:38.604565 22296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:38.610275 22296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44091,127.15.93.189:37363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.189.keytab
--principal=oryx/127.15.93.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:37363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.93.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
W20260927 16:28:38.610491 22296 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.93.190:44091,127.15.93.189:37363'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 16:28:38.611260 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:38.612207 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:38.620011 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.620010 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.620127 22296 server_base.cc:1034] running on GCE node
W20260927 16:28:38.620028 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.620792 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:38.621788 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:38.622967 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790526518622954 us; error 23 us; skew 500 ppm
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:38.625579 22296 init.cc:377] Logged in from keytab as oryx/127.15.93.189@KRBTEST.COM (short username oryx)
I20260927 16:28:38.626384 22296 webserver.cc:460] Webserver started at http://127.15.93.189:32971/ using document root <none> and password file <none>
I20260927 16:28:38.626907 22296 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.626992 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:38.627209 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:38.629315 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/instance:
uuid: "e47accc20b3a40638b37bfcfc2cf1493"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ec0cc5827125fccecc9cb810b20c1b41"
server_key_iv: "b73885bb43cf0fab5df1a1abfc3cb201"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.629837 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal/instance:
uuid: "e47accc20b3a40638b37bfcfc2cf1493"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ec0cc5827125fccecc9cb810b20c1b41"
server_key_iv: "b73885bb43cf0fab5df1a1abfc3cb201"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.633206 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:28:38.635330 22311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.636490 22296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:38.636634 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "e47accc20b3a40638b37bfcfc2cf1493"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ec0cc5827125fccecc9cb810b20c1b41"
server_key_iv: "b73885bb43cf0fab5df1a1abfc3cb201"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.636737 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.647738 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.650768 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:38.650959 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.657305 22296 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.189:37363
I20260927 16:28:38.657570 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.189:37363 every 8 connection(s)
I20260927 16:28:38.658252 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
I20260927 16:28:38.659651 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22296
I20260927 16:28:38.661087 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:38.665614 22364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.189@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:38.674600 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.661072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.15.93.189:37363)
0927 16:28:38.661495 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.661513 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.661554 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:28:38.662367 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.662387 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.662775 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.663137 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.663823 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.665309 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.666187 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.666819 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.666964 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.669779 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.669784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.669792 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.669816 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.671574 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.672063 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.672067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.672071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.672175 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.672478 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.672482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.672485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.672973 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.673200 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.673471 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.673762 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260927 16:28:38.676627 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.667468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.15.93.190:44091)
0927 16:28:38.667607 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.667612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.667876 (+ 264us) server_negotiation.cc:407] Connection header received
0927 16:28:38.668042 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.668045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.668096 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.668175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.668875 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.669435 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.670596 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.670963 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.671083 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.673439 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.673443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.673446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.673476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.674908 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.675346 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.675350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.675352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.675458 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.675735 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.675739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.675741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.675951 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.676059 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.676358 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.676471 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:28:38.676666 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.667434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41266)
0927 16:28:38.667760 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.667796 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.667934 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.668247 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.668254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.668566 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.668739 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.668751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.669573 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.669576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.670443 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.670453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.671085 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.671088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.671173 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.671688 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.671703 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.673306 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.675012 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.675021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.675027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.675243 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.675550 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.675554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.675556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.675644 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.676055 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.676061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.676169 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.676305 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.676472 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":114,"threads_started":1}
I20260927 16:28:38.677678 22364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.189@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
I20260927 16:28:38.685518 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.678104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35610)
0927 16:28:38.678251 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.678265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.678337 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.678562 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.678565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.678720 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.678792 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.678797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.679994 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.679997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.680491 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.680498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.680953 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.680956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.681063 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.681415 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.681425 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.682648 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.684166 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.684170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.684172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.684344 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.684599 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.684604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.684606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.684684 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.685019 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.685021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.685103 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.685209 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.685335 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 16:28:38.685523 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.678183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.15.93.189:37363)
0927 16:28:38.678274 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.678280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.678299 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:28:38.678429 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.678432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.678477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.678554 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.678894 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.679898 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.680599 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.680828 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.680938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.682779 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.682783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.682785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.682806 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.684044 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.684433 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.684439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.684441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.684511 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.684775 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.684778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.684779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.684928 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.685030 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.685263 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.685391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":40}
I20260927 16:28:38.688243 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Bootstrap starting.
I20260927 16:28:38.690557 22364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:38.691295 22364 log.cc:824] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:38.693065 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: No bootstrap required, opened a new log
I20260927 16:28:38.696180 22364 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } }
I20260927 16:28:38.696375 22364 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:38.696442 22364 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47accc20b3a40638b37bfcfc2cf1493, State: Initialized, Role: FOLLOWER
I20260927 16:28:38.696831 22364 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } }
I20260927 16:28:38.697355 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } } }
I20260927 16:28:38.697484 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:38.697947 22364 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:28:38.702075 22381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:38.702165 22381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:38.704085 22296 master_runner.cc:428] Detected that this master e47accc20b3a40638b37bfcfc2cf1493 is joining an existing cluster
I20260927 16:28:38.704159 22296 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.93.189:37363
I20260927 16:28:38.714545 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.706368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.15.93.190:44091)
0927 16:28:38.706517 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.706521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.706535 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:38.706642 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.706647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.706697 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.706771 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.707544 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.708102 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709347 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.709624 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709679 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.711241 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.711244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.711247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.711269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.712871 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713398 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.713403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.713405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.713486 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713760 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.713765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.713766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.713934 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.714036 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.714283 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.714389 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:28:38.714572 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.705929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41274)
0927 16:28:38.706450 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.706465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.706554 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.706895 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.706898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.707180 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.707405 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.707413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.708269 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.708271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.709173 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.709182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.709809 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.709908 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.710661 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.710675 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.711096 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.712997 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.713002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.713300 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.713581 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.713585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.713672 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.714026 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.714029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.714117 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.714223 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.714370 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":47,"spinlock_wait_cycles":47232,"thread_start_us":224,"threads_started":1}
I20260927 16:28:38.714658 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.706110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.15.93.189:37363)
0927 16:28:38.706208 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.706213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.706838 (+ 625us) server_negotiation.cc:407] Connection header received
0927 16:28:38.706908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.706911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.706955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.707038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.707625 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.708392 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709474 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.709809 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.711224 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.711228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.711231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.711254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.712883 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713390 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.713393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.713395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.713459 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713714 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.713716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.713718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.713886 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.713948 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.714254 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.714479 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":30}
I20260927 16:28:38.714890 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.706079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35612)
0927 16:28:38.706742 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.706756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.706835 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.707072 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.707076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.707280 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.707474 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.707482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.708505 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.708508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.709374 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.709382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.709920 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.709923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.710097 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.710529 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.710545 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.711086 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.712993 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.712996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.712998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.713295 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.713558 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.713560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.713562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.713630 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.714005 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.714010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.714104 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.714304 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.714589 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":77,"spinlock_wait_cycles":896,"thread_start_us":80,"threads_started":1}
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.190@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
I20260927 16:28:38.727612 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.718689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.15.93.189:37363)
0927 16:28:38.718800 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.718804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.719112 (+ 308us) server_negotiation.cc:407] Connection header received
0927 16:28:38.719279 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.719281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.719320 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.719431 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.720092 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.720529 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.721904 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.722132 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.722220 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.724411 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.724415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.724418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.724445 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.725762 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.726252 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.726256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.726258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.726332 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.726604 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.726607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.726609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.726871 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.726987 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.727319 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.727456 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:38.727690 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.718654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35624)
0927 16:28:38.718981 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.719034 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.719158 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.719458 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.719464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.719876 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.719987 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.719994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.720805 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.720809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.721780 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.721791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.722246 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.722249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.722356 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.722794 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.722809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.724283 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.725905 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.725915 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.725921 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.726154 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.726421 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.726425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.726427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.726511 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.726985 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.726992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.727112 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.727268 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.727494 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260927 16:28:38.730575 21832 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.93.189:37363: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493"
member_type: NON_VOTER
last_known_addr {
host: "127.15.93.189"
port: 37363
}
attrs {
promote: true
}
}
dest_uuid: "be0ed8ce3c0c44c0912336d08e3acc72"
cas_config_opid_index: -1
I20260927 16:28:38.731287 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } }
I20260927 16:28:38.733197 22391 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } }
I20260927 16:28:38.734028 22338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:38.735044 22391 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.93.189:37363
I20260927 16:28:38.735504 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } }
I20260927 16:28:38.735579 22338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 16:28:38.735611 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:38.736028 22296 master.cc:537] Master@127.15.93.189:37363 shutting down...
I20260927 16:28:38.736220 22393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.15.93.188@KRBTEST.COM; defaulting to no policy
W20260927 16:28:38.738263 21818 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 16:28:38.739120 21818 consensus_peers.cc:487] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 -> Peer e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189:37363): Couldn't send request to peer e47accc20b3a40638b37bfcfc2cf1493. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.188@KRBTEST.COM" created.
I20260927 16:28:38.740206 22296 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:38.740473 22296 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:38.740576 22296 tablet_replica.cc:330] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: stopping tablet replica
I20260927 16:28:38.754769 22296 master.cc:552] Master@127.15.93.189:37363 shutdown complete.
I20260927 16:28:38.754817 22296 master_runner.cc:305] Clearing existing system tablet
I20260927 16:28:38.755043 22296 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.755105 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.188.keytab.
Entry for principal oryx/127.15.93.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.188.keytab.
I20260927 16:28:38.756181 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:42007
--webserver_interface=127.15.93.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.188.keytab
--principal=oryx/127.15.93.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:28:38.756482 22296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:38.763793 22399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.764020 22296 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.764130 22296 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 16:28:38.764161 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "e47accc20b3a40638b37bfcfc2cf1493"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ec0cc5827125fccecc9cb810b20c1b41"
server_key_iv: "b73885bb43cf0fab5df1a1abfc3cb201"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.766294 22296 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:38.768903 22296 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:28:38.769045 22296 log.cc:1196] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 16:28:38.769331 22296 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Deleting consensus metadata
I20260927 16:28:38.769520 22296 master_runner.cc:315] Copying system tablet from 127.15.93.190:44091
I20260927 16:28:38.770721 22296 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: tablet copy: Beginning tablet copy session from remote peer at address 127.15.93.190:44091
I20260927 16:28:38.779461 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.771514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.15.93.190:44091)
0927 16:28:38.771637 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.771642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.772052 (+ 410us) server_negotiation.cc:407] Connection header received
0927 16:28:38.772219 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.772223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.772273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.772324 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.772913 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.773765 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.774938 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.775169 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.775285 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.776375 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.776379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.776381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.776406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.777872 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.778291 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.778296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.778298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.778379 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.778662 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.778666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.778667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.778836 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.778941 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.779184 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.779289 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:28:38.779482 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.771723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41284)
0927 16:28:38.771975 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.771988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.772121 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.772460 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.772463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.772677 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.772791 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.772797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.773933 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.773936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.774811 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.774822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.775285 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.775288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.775410 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.775921 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.775939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.776239 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.778000 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.778003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.778005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.778197 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.778475 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.778478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.778480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.778559 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.778931 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.778934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.779028 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.779132 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.779260 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":55,"threads_started":1}
I20260927 16:28:38.780223 21852 tablet_copy_service.cc:140] P be0ed8ce3c0c44c0912336d08e3acc72: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e47accc20b3a40638b37bfcfc2cf1493 ({username='oryx', principal='oryx/127.15.93.189@KRBTEST.COM'} at 127.0.0.1:41284)
I20260927 16:28:38.780385 21852 tablet_copy_service.cc:161] P be0ed8ce3c0c44c0912336d08e3acc72: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e47accc20b3a40638b37bfcfc2cf1493 at {username='oryx', principal='oryx/127.15.93.189@KRBTEST.COM'} at 127.0.0.1:41284: session id = e47accc20b3a40638b37bfcfc2cf1493-00000000000000000000000000000000
I20260927 16:28:38.781827 21852 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:28:38.783187 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:28:38.784888 22296 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: tablet copy: Starting download of 0 data blocks...
I20260927 16:28:38.785101 22296 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: tablet copy: Starting download of 1 WAL segments...
I20260927 16:28:38.786501 22296 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:28:38.787889 21852 tablet_copy_service.cc:342] P be0ed8ce3c0c44c0912336d08e3acc72: Request end of tablet copy session e47accc20b3a40638b37bfcfc2cf1493-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.93.189@KRBTEST.COM'} at 127.0.0.1:41284
I20260927 16:28:38.788012 21852 tablet_copy_service.cc:433] P be0ed8ce3c0c44c0912336d08e3acc72: ending tablet copy session e47accc20b3a40638b37bfcfc2cf1493-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e47accc20b3a40638b37bfcfc2cf1493
I20260927 16:28:38.794385 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:38.795900 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.796010 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.796164 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.796739 22296 server_base.cc:1034] running on GCE node
I20260927 16:28:38.796891 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:38.797183 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:38.798358 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790526518798346 us; error 27 us; skew 500 ppm
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:38.799842 22296 init.cc:377] Logged in from keytab as oryx/127.15.93.189@KRBTEST.COM (short username oryx)
I20260927 16:28:38.800292 22296 webserver.cc:460] Webserver started at http://127.15.93.189:46765/ using document root <none> and password file <none>
I20260927 16:28:38.800515 22296 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.800570 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:38.801587 22296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:38.802450 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.802675 22296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:38.802765 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "e47accc20b3a40638b37bfcfc2cf1493"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ec0cc5827125fccecc9cb810b20c1b41"
server_key_iv: "b73885bb43cf0fab5df1a1abfc3cb201"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.802847 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.813829 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.815799 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.820648 22296 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.189:37363
I20260927 16:28:38.821218 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
I20260927 16:28:38.821611 22468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.189:37363 every 8 connection(s)
I20260927 16:28:38.822638 22469 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:28:38.823480 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Bootstrap starting.
I20260927 16:28:38.831264 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:28:38.831588 22469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: Bootstrap complete.
I20260927 16:28:38.832093 22469 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } }
I20260927 16:28:38.832235 22469 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e47accc20b3a40638b37bfcfc2cf1493, State: Initialized, Role: LEARNER
I20260927 16:28:38.832371 22469 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } }
I20260927 16:28:38.832705 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } }
I20260927 16:28:38.832852 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:28:38.832863 22469 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:38.836206 22483 catalog_manager.cc:1259] Loaded cluster ID: 36bf8cf5459a4de38a75de51df30a930
I20260927 16:28:38.836256 22483 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: loading cluster ID for follower catalog manager: success
I20260927 16:28:38.837738 22483 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: acquiring CA information for follower catalog manager: success
I20260927 16:28:38.838776 22483 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:28:38.905344 22398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:38.905591 22398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:38.905629 22398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:38.909740 22398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:38.909790 22398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:38.909809 22398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:38.909825 22398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:38.909855 22398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:38.914949 22398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.188.keytab
--principal=oryx/127.15.93.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.93.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:38.915937 22398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:38.916846 22398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:38.924929 22489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:38.924909 22491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.925071 22398 server_base.cc:1034] running on GCE node
W20260927 16:28:38.925257 22488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:38.925806 22398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:38.926812 22398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:38.927968 22398 hybrid_clock.cc:648] HybridClock initialized: now 1790526518927954 us; error 24 us; skew 500 ppm
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:38.930642 22398 init.cc:377] Logged in from keytab as oryx/127.15.93.188@KRBTEST.COM (short username oryx)
I20260927 16:28:38.931450 22398 webserver.cc:460] Webserver started at http://127.15.93.188:34663/ using document root <none> and password file <none>
I20260927 16:28:38.931938 22398 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:38.931993 22398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:38.932163 22398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:38.934303 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/instance:
uuid: "103c04f460ca4c47a9bac35eaf77dc54"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "e61e9839784c15e6323ace6a65337dc9"
server_key_iv: "1729fef40dc943309382c5bf4881dc3c"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.934901 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal/instance:
uuid: "103c04f460ca4c47a9bac35eaf77dc54"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "e61e9839784c15e6323ace6a65337dc9"
server_key_iv: "1729fef40dc943309382c5bf4881dc3c"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.938342 22398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:38.940656 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:38.941816 22398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:38.941944 22398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "103c04f460ca4c47a9bac35eaf77dc54"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "e61e9839784c15e6323ace6a65337dc9"
server_key_iv: "1729fef40dc943309382c5bf4881dc3c"
server_key_version: "encryptionkey@0"
I20260927 16:28:38.942049 22398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:38.967991 22398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:38.970690 22398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:38.970880 22398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:38.978116 22398 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.188:42007
I20260927 16:28:38.978389 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.188:42007 every 8 connection(s)
I20260927 16:28:38.979121 22398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
I20260927 16:28:38.981982 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:38.983052 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22398
I20260927 16:28:38.987421 22551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
Sep 27 16:28:38 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.188@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:38.998157 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.984532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.15.93.188:42007)
0927 16:28:38.984896 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.984910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.984945 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:28:38.985859 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.985877 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.986191 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.986458 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.987325 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.988434 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.989495 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.990145 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.990313 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.992959 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.992965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.992977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.993013 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.995054 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.995536 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.995541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.995547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.995701 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.996037 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.996040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.996041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.996547 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.996689 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.997076 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.997383 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":120,"threads_started":1}
I20260927 16:28:38.998917 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.989015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41288)
0927 16:28:38.989406 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:38.989451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:38.989598 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:38.989871 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:38.989875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:38.990328 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:38.990449 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.990455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.991579 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.991582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.992205 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:38.992214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.992820 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.992824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:38.992935 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.993494 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:38.993521 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:38.995206 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:38.997321 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.997329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.997334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.997562 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.997857 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:38.997861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:38.997863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:38.997950 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:38.998456 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:38.998462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:38.998549 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 16:28:38.998660 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 16:28:38.998787 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":169,"threads_started":1}
I20260927 16:28:38.998992 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:38.989148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.15.93.190:44091)
0927 16:28:38.989267 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:38.989271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:38.989568 (+ 297us) server_negotiation.cc:407] Connection header received
0927 16:28:38.989689 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:38.989692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:38.989742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:38.989823 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:38.990605 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.991428 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.992360 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:38.992675 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:38.992786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:38.995345 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:38.995348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:38.995351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:38.995404 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:38.997188 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.997675 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.997679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.997680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.997765 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:38.998069 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:38.998072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:38.998073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:38.998294 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:38.998397 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:38.998747 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 16:28:38.998858 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:28:39.000296 22551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.188@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
I20260927 16:28:39.008675 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.000751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35638)
0927 16:28:39.001154 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.001169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.001253 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.001492 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.001495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.001640 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.001737 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.001743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.002611 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.002613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.003354 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.003363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.003889 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.003891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.003996 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.004385 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.004395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.005514 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.007180 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.007186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.007188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.007446 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.007735 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.007739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.007741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.007824 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.008196 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.008200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.008289 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.008394 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.008518 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260927 16:28:39.008740 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.000885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.15.93.189:37363)
0927 16:28:39.001148 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.001159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.001226 (+ 67us) server_negotiation.cc:407] Connection header received
0927 16:28:39.001334 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.001336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.001375 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.001447 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.001843 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.002498 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.003540 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.003779 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.003858 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.005675 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.005678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.005680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.005703 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.007063 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.007549 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.007554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.007556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.007641 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.007918 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.007922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.007923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.008095 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.008195 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.008467 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.008582 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260927 16:28:39.009531 22551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526518, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.188@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
I20260927 16:28:39.016969 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.009934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50528)
0927 16:28:39.010035 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.010049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.010123 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.010315 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.010317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.010433 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.010494 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.010498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.011552 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.011554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.012030 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.012037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.012437 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.012439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.012493 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.012852 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.012862 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.013961 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.015645 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.015648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.015650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.015826 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.016082 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.016087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.016089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.016164 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.016495 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.016497 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.016570 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.016688 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.016811 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:28:39.016994 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.010024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.15.93.188:42007)
0927 16:28:39.010125 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.010128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.010139 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:39.010202 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.010203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.010234 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.010289 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.010581 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.011436 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.012124 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.012349 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.012425 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.014096 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.014099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.014101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.014119 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.015524 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.015917 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.015923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.015925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.015994 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.016254 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.016257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.016258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.016405 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.016504 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.016730 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.016863 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":40}
I20260927 16:28:39.019897 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Bootstrap starting.
I20260927 16:28:39.021920 22551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:39.022558 22551 log.cc:824] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:39.024302 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: No bootstrap required, opened a new log
I20260927 16:28:39.026999 22551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } }
I20260927 16:28:39.027197 22551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:39.027230 22551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c04f460ca4c47a9bac35eaf77dc54, State: Initialized, Role: FOLLOWER
I20260927 16:28:39.027638 22551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } }
I20260927 16:28:39.028055 22557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } } }
I20260927 16:28:39.028188 22557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:39.028702 22551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:28:39.033237 22568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:39.033310 22568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:39.036279 22023 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.036581 22398 master_runner.cc:428] Detected that this master 103c04f460ca4c47a9bac35eaf77dc54 is joining an existing cluster
I20260927 16:28:39.036635 22398 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.93.188:42007
I20260927 16:28:39.048611 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.039422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41304)
0927 16:28:39.040056 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.040068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.040192 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.041699 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.041701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.041914 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.042250 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.042258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.043264 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.043266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.043892 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.043900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.044360 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.044364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.044460 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.044888 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.044902 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.045317 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.047093 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.047096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.047099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.047350 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.047672 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.047675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.047677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.047767 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.048117 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.048120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.048212 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.048305 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.048428 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":17,"thread_start_us":98,"threads_started":1}
I20260927 16:28:39.048765 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.039471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.15.93.190:44091)
0927 16:28:39.039598 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.039604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.040126 (+ 522us) server_negotiation.cc:407] Connection header received
0927 16:28:39.040288 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.040291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.041471 (+ 1180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.041542 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.042386 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.043144 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.044012 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.044237 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.044362 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.045461 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.045464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.045466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.045487 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.046972 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.047496 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.047499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.047500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.047581 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.047860 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.047863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.047864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.048034 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.048106 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.048463 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.048591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:39.051445 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21894
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526517, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.190@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
I20260927 16:28:39.059746 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:45823
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:44091
--webserver_port=33909
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:28:39.072592 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.039014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50534)
0927 16:28:39.039727 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.039737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.039845 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.040148 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.040151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.040343 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.040451 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.040458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.041467 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.041470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.042450 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.042460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.043099 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.043102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.043207 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.043749 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.043764 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.044186 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.070853 (+ 26667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.070856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.070858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.071101 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.071356 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.071361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.071456 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.071935 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.071937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.072029 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.072110 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.072250 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":77,"thread_start_us":82,"threads_started":1}
I20260927 16:28:39.072973 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.038625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.15.93.189:37363)
0927 16:28:39.038765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.038770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.038956 (+ 186us) server_negotiation.cc:407] Connection header received
0927 16:28:39.039138 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.039141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.039190 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.039242 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.039896 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.041103 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.042596 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.042942 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.043009 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.069318 (+ 26309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.069323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.069326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.069356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.071020 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071543 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.071547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.071549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.071625 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071916 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.071920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.071921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.072124 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.072208 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.072699 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.072824 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:28:39.073344 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.038573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35654)
0927 16:28:39.038876 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.038891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.039021 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.039423 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.039426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.039647 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.039790 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.039797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.041373 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.041376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.042479 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.042487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.043166 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.043172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.043326 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.068148 (+ 24822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.068166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.069106 (+ 940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.071151 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.071157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.071433 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.071731 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.071736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.071821 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.072249 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.072252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.072349 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.072434 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.072567 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":70,"spinlock_wait_cycles":2176,"thread_start_us":131,"threads_started":1}
I20260927 16:28:39.073839 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.039045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.15.93.188:42007)
0927 16:28:39.039147 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.039151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.039809 (+ 658us) server_negotiation.cc:407] Connection header received
0927 16:28:39.039948 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.039951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.039991 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.040056 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.040564 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.041336 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.042568 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.042891 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.042959 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.044320 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.044326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.044331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.044357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.070723 (+ 26366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071202 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.071205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.071207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.071275 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.071544 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.071546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.071548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.071818 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.071904 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.073591 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.073703 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:28:39.077368 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.052338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50538)
0927 16:28:39.052442 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.052455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.052527 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.053147 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.053153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.053349 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.053492 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.053499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.054565 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.054571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.055601 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.055614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.056177 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.056184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.056310 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.056795 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.056815 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.058404 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.075765 (+ 17361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.075769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.075772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.076050 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.076346 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.076349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.076351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.076448 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.076843 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.076846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.076951 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.077057 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.077212 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 16:28:39.077720 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.052654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.15.93.188:42007)
0927 16:28:39.052894 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.052902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.052914 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:39.052972 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.052975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.053019 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.053106 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.053603 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.054409 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.055715 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.056057 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.056156 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.058544 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.058551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.058557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.058587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.075628 (+ 17041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.076152 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.076155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.076157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.076245 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.076543 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.076546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.076550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.076746 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.076838 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.077130 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.077240 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260927 16:28:39.078578 21832 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.93.188:42007: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54"
member_type: NON_VOTER
last_known_addr {
host: "127.15.93.188"
port: 42007
}
attrs {
promote: true
}
}
dest_uuid: "be0ed8ce3c0c44c0912336d08e3acc72"
cas_config_opid_index: 5
I20260927 16:28:39.079195 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } }
I20260927 16:28:39.080307 22392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.081338 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:39.081722 22392 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.93.188:42007
I20260927 16:28:39.081880 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.081969 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.082389 22398 master.cc:537] Master@127.15.93.188:42007 shutting down...
I20260927 16:28:39.082688 22525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:28:39.083312 22393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:28:39.085417 21818 consensus_peers.cc:487] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 -> Peer 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188:42007): Couldn't send request to peer 103c04f460ca4c47a9bac35eaf77dc54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 16:28:39.087330 22398 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:39.087610 22398 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:39.087700 22398 tablet_replica.cc:330] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: stopping tablet replica
I20260927 16:28:39.091295 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.080525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35658)
0927 16:28:39.086555 (+ 6030us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.086570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.086650 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.087137 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.087140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.087161 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.087270 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.087279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.088401 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.088404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.089806 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.089817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.090894 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.090897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.090989 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.091031 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.091100 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.091171 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5947}
I20260927 16:28:39.091837 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.080740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.15.93.189:37363)
0927 16:28:39.086407 (+ 5667us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.086411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.086834 (+ 423us) server_negotiation.cc:407] Connection header received
0927 16:28:39.086882 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.086885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.086939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.087020 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.087409 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.088194 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.089941 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.090585 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.090693 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.090833 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.091642 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.091728 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5582}
I20260927 16:28:39.092468 22443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 LEARNER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 16:28:39.092890 22393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 16:28:39.094740 22443 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.096824 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0ed8ce3c0c44c0912336d08e3acc72. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.096932 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:28:39.096851 22580 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: NON_VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: true } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.096987 22580 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:28:39.097606 22391 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: attempting to promote NON_VOTER e47accc20b3a40638b37bfcfc2cf1493 to VOTER
I20260927 16:28:39.098035 22391 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } }
I20260927 16:28:39.099217 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.098947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50542)
0927 16:28:39.099078 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.099150 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 16:28:39.099529 22443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 LEARNER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 16:28:39.099923 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.099725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50552)
0927 16:28:39.099817 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.099867 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 16:28:39.100090 21818 consensus_peers.cc:487] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 -> Peer 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188:42007): Couldn't send request to peer 103c04f460ca4c47a9bac35eaf77dc54. Status: Network error: Client connection negotiation failed: client connection to 127.15.93.188:42007: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:28:39.100232 22393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 16:28:39.102417 22392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.102840 22442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.103801 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.103893 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.104003 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.104063 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.104341 22580 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.104485 22580 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:39.105755 22398 master.cc:552] Master@127.15.93.188:42007 shutdown complete.
I20260927 16:28:39.105808 22398 master_runner.cc:305] Clearing existing system tablet
I20260927 16:28:39.106051 22398 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:39.106106 22398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:39.107609 22398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:39.108498 22582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.108773 22398 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:39.108894 22398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:39.108927 22398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "103c04f460ca4c47a9bac35eaf77dc54"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "e61e9839784c15e6323ace6a65337dc9"
server_key_iv: "1729fef40dc943309382c5bf4881dc3c"
server_key_version: "encryptionkey@0"
I20260927 16:28:39.110826 22398 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:39.113095 22398 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 16:28:39.113181 22398 log.cc:1196] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 16:28:39.113413 22398 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Deleting consensus metadata
I20260927 16:28:39.113579 22398 master_runner.cc:315] Copying system tablet from 127.15.93.190:44091
I20260927 16:28:39.114672 22398 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: tablet copy: Beginning tablet copy session from remote peer at address 127.15.93.190:44091
I20260927 16:28:39.123737 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.115224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.15.93.190:44091)
0927 16:28:39.115335 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.115339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.115531 (+ 192us) server_negotiation.cc:407] Connection header received
0927 16:28:39.115634 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.115638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.115689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.115776 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.116258 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.117196 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.118333 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.118653 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.118755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.120012 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.120016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.120019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.120045 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.122028 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.122494 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.122497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.122499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.122588 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.122874 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.122877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.122878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.123091 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.123187 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.123486 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.123598 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:28:39.123749 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.115141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41312)
0927 16:28:39.115434 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.115449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.115555 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.115819 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.115822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.116022 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.116125 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.116131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.117314 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.117317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.118208 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.118218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.118768 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.118772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.118861 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.119428 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.119443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.119861 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.122140 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.122143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.122145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.122392 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.122689 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.122692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.122694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.122782 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.123190 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.123193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.123285 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.123413 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.123545 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260927 16:28:39.125488 21852 tablet_copy_service.cc:140] P be0ed8ce3c0c44c0912336d08e3acc72: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 103c04f460ca4c47a9bac35eaf77dc54 ({username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:41312)
I20260927 16:28:39.125576 21852 tablet_copy_service.cc:161] P be0ed8ce3c0c44c0912336d08e3acc72: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 103c04f460ca4c47a9bac35eaf77dc54 at {username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:41312: session id = 103c04f460ca4c47a9bac35eaf77dc54-00000000000000000000000000000000
I20260927 16:28:39.126691 21852 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 16:28:39.127874 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 16:28:39.130198 22398 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: tablet copy: Starting download of 0 data blocks...
I20260927 16:28:39.130381 22398 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: tablet copy: Starting download of 1 WAL segments...
I20260927 16:28:39.131662 22398 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 16:28:39.133288 21852 tablet_copy_service.cc:342] P be0ed8ce3c0c44c0912336d08e3acc72: Request end of tablet copy session 103c04f460ca4c47a9bac35eaf77dc54-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:41312
I20260927 16:28:39.133373 21852 tablet_copy_service.cc:433] P be0ed8ce3c0c44c0912336d08e3acc72: ending tablet copy session 103c04f460ca4c47a9bac35eaf77dc54-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 103c04f460ca4c47a9bac35eaf77dc54
I20260927 16:28:39.140417 22398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:39.141975 22589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.141975 22592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.142187 22590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:39.142633 22398 server_base.cc:1034] running on GCE node
I20260927 16:28:39.142781 22398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:39.143039 22398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:39.144186 22398 hybrid_clock.cc:648] HybridClock initialized: now 1790526519144173 us; error 27 us; skew 500 ppm
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:39.145740 22398 init.cc:377] Logged in from keytab as oryx/127.15.93.188@KRBTEST.COM (short username oryx)
I20260927 16:28:39.146176 22398 webserver.cc:460] Webserver started at http://127.15.93.188:44643/ using document root <none> and password file <none>
I20260927 16:28:39.146355 22398 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:39.146400 22398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:39.147722 22398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:39.148705 22599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.148960 22398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:39.149032 22398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "103c04f460ca4c47a9bac35eaf77dc54"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "e61e9839784c15e6323ace6a65337dc9"
server_key_iv: "1729fef40dc943309382c5bf4881dc3c"
server_key_version: "encryptionkey@0"
I20260927 16:28:39.149112 22398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:39.158129 22398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:39.160307 22398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:39.165165 22398 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.188:42007
I20260927 16:28:39.165198 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.188:42007 every 8 connection(s)
I20260927 16:28:39.165741 22398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
I20260927 16:28:39.168066 22652 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:28:39.169279 22652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Bootstrap starting.
I20260927 16:28:39.180804 22652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:28:39.181123 22652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Bootstrap complete.
I20260927 16:28:39.181789 22652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } }
I20260927 16:28:39.181942 22652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 103c04f460ca4c47a9bac35eaf77dc54, State: Initialized, Role: LEARNER
I20260927 16:28:39.182057 22652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } }
I20260927 16:28:39.182404 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.182524 22652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:39.182605 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:28:39.187037 22666 catalog_manager.cc:1259] Loaded cluster ID: 36bf8cf5459a4de38a75de51df30a930
I20260927 16:28:39.187098 22666 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: loading cluster ID for follower catalog manager: success
I20260927 16:28:39.189237 22666 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: acquiring CA information for follower catalog manager: success
I20260927 16:28:39.190562 22666 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:28:39.203131 22578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:39.203413 22578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:39.203492 22578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:39.207552 22578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:39.207623 22578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:39.207711 22578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:39.213125 22578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:45823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=33909
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:39.214130 22578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:39.215116 22578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:39.224411 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.224406 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.224419 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:39.224924 22578 server_base.cc:1034] running on GCE node
I20260927 16:28:39.225311 22578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:39.225909 22578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:39.227101 22578 hybrid_clock.cc:648] HybridClock initialized: now 1790526519227092 us; error 32 us; skew 500 ppm
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:39.229826 22578 init.cc:377] Logged in from keytab as oryx/127.15.93.129@KRBTEST.COM (short username oryx)
I20260927 16:28:39.230592 22578 webserver.cc:460] Webserver started at http://127.15.93.129:33909/ using document root <none> and password file <none>
I20260927 16:28:39.231130 22578 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:39.231230 22578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:39.235154 22578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 16:28:39.237167 22681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.238359 22578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:39.238521 22578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "b09bcd3a45014e7998b2d0ca225ffa69"
format_stamp: "Formatted at 2026-09-27 16:28:37 on dist-test-slave-grzf"
server_key: "9bc74b8364999e3c703d7f0a5bbb3c36"
server_key_iv: "0c16c1514bc2c18fbdecd12b0e93f7ab"
server_key_version: "encryptionkey@0"
I20260927 16:28:39.239043 22578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:39.241853 22157 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.254961 22578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:39.257834 22578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:39.258030 22578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:39.258591 22578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:39.259460 22578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:39.259523 22578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.259584 22578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:39.259634 22578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.267977 22578 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:45823
I20260927 16:28:39.267999 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:45823 every 8 connection(s)
I20260927 16:28:39.268936 22578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:39.279167 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22578
I20260927 16:28:39.279409 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22028
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
I20260927 16:28:39.284384 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.270658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:41655 (local address 127.15.93.190:44091)
0927 16:28:39.270795 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.270799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.271674 (+ 875us) server_negotiation.cc:407] Connection header received
0927 16:28:39.272833 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.272837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.272897 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.272976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.274355 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275168 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.276611 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.276931 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.277008 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.279618 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.279622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.279626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.279654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.281478 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.282140 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.282146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.282149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.282250 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.282773 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.282779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.282782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.283032 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.283128 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.284042 (+ 914us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.284199 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:28:39.285497 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.270754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:42369 (local address 127.15.93.189:37363)
0927 16:28:39.270898 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.270902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.271677 (+ 775us) server_negotiation.cc:407] Connection header received
0927 16:28:39.272942 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.272948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.273009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.273085 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.274356 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275175 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.276396 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.276702 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.276801 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.281261 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.281268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.281274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.281309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.283089 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.283643 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.283649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.283654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.283740 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.284119 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.284123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.284124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.284371 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.284457 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.285134 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.285264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 16:28:39.285998 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.272032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.15.93.129:53847)
0927 16:28:39.272286 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.272300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.272647 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.273294 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.273297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.273681 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.274199 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.274212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275177 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.275182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.276494 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.276505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.277542 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.277547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.277631 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.278009 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.278026 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.281470 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.284075 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.284078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.284080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.284314 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.284616 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.284621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.284623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.284715 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.285153 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.285156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.285236 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.285328 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.285445 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":147,"thread_start_us":81,"threads_started":1}
I20260927 16:28:39.286264 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.271058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:41655)
0927 16:28:39.271517 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.271567 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.272648 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.273160 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.273169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.273720 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.274214 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.274220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275261 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.275263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.276500 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.276505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.277037 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.277040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.277136 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.277825 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.277841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.279429 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.281653 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.281661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.281677 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.282017 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.282370 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.282425 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.282428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.282659 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.283215 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.283223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.283712 (+ 489us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.284542 (+ 830us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.284979 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":274,"spinlock_wait_cycles":19200,"thread_start_us":126,"threads_started":1}
I20260927 16:28:39.286715 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.272171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:53847 (local address 127.15.93.188:42007)
0927 16:28:39.272433 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.272447 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.272463 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:39.273108 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.273111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.273173 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.273278 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.274339 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275040 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.277095 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.277410 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.277520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.281878 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.281884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.281887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.281918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.283932 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.284425 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.284429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.284430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.284509 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.284820 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.284823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.284825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.285028 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.285113 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.286395 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.286525 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260927 16:28:39.287248 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.271058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.15.93.129:42369)
0927 16:28:39.271516 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.271567 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.272635 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.273170 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.273173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.273724 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.274211 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.274220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.275308 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.275311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.276272 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.276281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.276831 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.276833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.276966 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.277637 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.277657 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.281091 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.283219 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.283225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.283230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.283517 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.283852 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.283858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.283863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.283978 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.284817 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.284823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.284931 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.285887 (+ 956us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.286725 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":185,"spinlock_wait_cycles":48384,"thread_start_us":148,"threads_started":1}
I20260927 16:28:39.287402 22796 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:39.287771 22796 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.288219 22797 heartbeater.cc:347] Connected to a master server at 127.15.93.189:37363
I20260927 16:28:39.288314 22797 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.288362 22796 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
I20260927 16:28:39.288491 22797 heartbeater.cc:511] Master 127.15.93.189:37363 requested a full tablet report, sending...
I20260927 16:28:39.287418 22795 heartbeater.cc:347] Connected to a master server at 127.15.93.188:42007
I20260927 16:28:39.288901 22795 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.289080 22795 heartbeater.cc:511] Master 127.15.93.188:42007 requested a full tablet report, sending...
I20260927 16:28:39.289196 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:43297
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:44091
--webserver_port=44897
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:28:39.289534 21832 ts_manager.cc:194] Re-registered known tserver with Master: b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823)
I20260927 16:28:39.290355 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:41655
I20260927 16:28:39.291213 22616 ts_manager.cc:194] Registered new tserver with Master: b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823)
I20260927 16:28:39.291213 22433 ts_manager.cc:194] Registered new tserver with Master: b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823)
W20260927 16:28:39.400000 22803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:39.400245 22803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:39.400288 22803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:39.404331 22803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:39.404382 22803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:39.404438 22803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:39.409505 22803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:43297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=44897
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:39.410431 22803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:39.411391 22803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:39.420819 22808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.420954 22811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.420814 22809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:39.421068 22803 server_base.cc:1034] running on GCE node
I20260927 16:28:39.421617 22803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:39.422202 22803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:39.423419 22803 hybrid_clock.cc:648] HybridClock initialized: now 1790526519423393 us; error 34 us; skew 500 ppm
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:39.425985 22803 init.cc:377] Logged in from keytab as oryx/127.15.93.130@KRBTEST.COM (short username oryx)
I20260927 16:28:39.426813 22803 webserver.cc:460] Webserver started at http://127.15.93.130:44897/ using document root <none> and password file <none>
I20260927 16:28:39.427345 22803 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:39.427498 22803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:39.431154 22803 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:39.433030 22818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.434392 22803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 16:28:39.434516 22803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "72df7b22a1d94e02b9dca37e93fc30ab"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "7f5ea379cbf20909d9153dc16ed9cd7c"
server_key_iv: "bb140a40c1153bafa728eda44836c183"
server_key_version: "encryptionkey@0"
I20260927 16:28:39.435062 22803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:39.463945 22291 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.481036 22803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:39.484082 22803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:39.484290 22803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:39.484889 22803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:39.485715 22803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:39.485782 22803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.485845 22803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:39.485893 22803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.495041 22803 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:43297
I20260927 16:28:39.495070 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:43297 every 8 connection(s)
I20260927 16:28:39.496028 22803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:39.496879 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22803
I20260927 16:28:39.497073 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22162
I20260927 16:28:39.504577 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:44169
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:44091
--webserver_port=34429
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:39.514591 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.497883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:32907 (local address 127.15.93.188:42007)
0927 16:28:39.498010 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.498014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.499459 (+ 1445us) server_negotiation.cc:407] Connection header received
0927 16:28:39.500725 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.500731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.500796 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.500855 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.503119 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.504058 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.505733 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506120 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506199 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.510046 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.510053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.510058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.510095 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.511870 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.512629 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.512633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.512635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.512721 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.513154 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.513160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.513166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.513408 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.513493 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.514230 (+ 737us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.514369 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:28:39.515693 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.499410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.15.93.130:54283)
0927 16:28:39.500262 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.500275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.500680 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.501401 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.501410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.502652 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.503089 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.503098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.505199 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.505205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.506068 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.506079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506828 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.507481 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.508021 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.508033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.509857 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.512443 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.512449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.512452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.512733 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.513077 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.513083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.513086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.513184 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.513722 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.513728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.513918 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.514130 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.514516 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":242,"spinlock_wait_cycles":6528,"thread_start_us":72,"threads_started":1}
I20260927 16:28:39.516292 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.499817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:35583 (local address 127.15.93.190:44091)
0927 16:28:39.499982 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.499989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.500008 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:28:39.500909 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.500915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.500978 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.501036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.503216 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.503859 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506123 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506433 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.512665 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.512671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.512676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.512707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.514389 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.514982 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.514985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.514987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.515075 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.515394 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.515398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.515400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.515597 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.515677 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.516030 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.516148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:28:39.516460 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.499959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:54283 (local address 127.15.93.189:37363)
0927 16:28:39.500078 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.500082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.500362 (+ 280us) server_negotiation.cc:407] Connection header received
0927 16:28:39.501001 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.501007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.501064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.501144 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.504454 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.505082 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506171 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506565 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506675 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.510436 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.510443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.510449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.510486 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.512300 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.512859 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.512866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.512871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.512961 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.513301 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.513306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.513310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.513554 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.513670 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.516111 (+ 2441us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.516246 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:28:39.516837 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.498222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:35583)
0927 16:28:39.499351 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.499366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.500496 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.501401 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.501410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.502049 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.502948 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.502968 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.504047 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.504050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.505991 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.506001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506642 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.506735 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.507366 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.507400 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.512508 (+ 5108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.514532 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.514535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.514536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.514875 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.515187 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.515189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.515191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.515267 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.515757 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.515760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.515862 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.516477 (+ 615us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.516644 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":927,"spinlock_wait_cycles":36352,"thread_start_us":159,"threads_started":1}
I20260927 16:28:39.517421 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.498222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.15.93.130:32907)
0927 16:28:39.499288 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.499342 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.500477 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.501401 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.501411 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.502372 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.502948 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.502969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.504242 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.504246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.505570 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.505584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.506383 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.506387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.506575 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.507211 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.507227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.509857 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.512006 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.512012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.512027 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.512506 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.512840 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.512845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.512846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.513020 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.513580 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.513588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.513919 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.514956 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.515087 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":238,"spinlock_wait_cycles":20992,"thread_start_us":653,"threads_started":1}
I20260927 16:28:39.517552 22933 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:39.517602 22934 heartbeater.cc:347] Connected to a master server at 127.15.93.189:37363
I20260927 16:28:39.517884 22933 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.517886 22934 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.518460 22932 heartbeater.cc:347] Connected to a master server at 127.15.93.188:42007
I20260927 16:28:39.518466 22933 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
I20260927 16:28:39.518585 22932 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.518730 22932 heartbeater.cc:511] Master 127.15.93.188:42007 requested a full tablet report, sending...
I20260927 16:28:39.518488 22934 heartbeater.cc:511] Master 127.15.93.189:37363 requested a full tablet report, sending...
I20260927 16:28:39.519642 21832 ts_manager.cc:194] Re-registered known tserver with Master: 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:39.519815 22616 ts_manager.cc:194] Registered new tserver with Master: 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:39.519815 22433 ts_manager.cc:194] Registered new tserver with Master: 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:39.520356 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:35583
W20260927 16:28:39.617233 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:39.617482 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:39.617522 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:39.621447 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:39.621490 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:39.621583 22939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:39.626473 22939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:44169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=34429
--tserver_master_addrs=127.15.93.190:44091,127.15.93.189:37363,127.15.93.188:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:39.627396 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:39.628362 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:28:39.638327 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.633942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.15.93.188:42007)
0927 16:28:39.634101 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.634107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.634332 (+ 225us) server_negotiation.cc:407] Connection header received
0927 16:28:39.634513 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.634520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.634592 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.634699 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.634942 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.635512 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.636897 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.637520 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.637635 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.637847 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.638082 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.638165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 16:28:39.638332 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.633847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50556)
0927 16:28:39.634239 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.634254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.634379 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.634720 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.634722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.634740 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.634846 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.634851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.635608 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.635611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.636789 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.636801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.637697 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.637702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.637922 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.637973 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.638050 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.638172 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":54,"threads_started":1}
W20260927 16:28:39.638324 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.638324 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:39.638756 22948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:39.638913 22626 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 16:28:39.639201 22939 server_base.cc:1034] running on GCE node
I20260927 16:28:39.639345 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0ed8ce3c0c44c0912336d08e3acc72. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.639513 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: LEARNER
I20260927 16:28:39.639688 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:39.640293 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:39.641451 22939 hybrid_clock.cc:648] HybridClock initialized: now 1790526519641427 us; error 32 us; skew 500 ppm
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:39.644114 22939 init.cc:377] Logged in from keytab as oryx/127.15.93.131@KRBTEST.COM (short username oryx)
I20260927 16:28:39.644892 22939 webserver.cc:460] Webserver started at http://127.15.93.131:34429/ using document root <none> and password file <none>
I20260927 16:28:39.645422 22939 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:39.645527 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:39.649221 22939 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:39.651139 22956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.652271 22939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:39.652396 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "18df5caabaed4b89903d368d5f8bdee3"
format_stamp: "Formatted at 2026-09-27 16:28:38 on dist-test-slave-grzf"
server_key: "ee199b4fa4fab32346fe8e0b32d93a38"
server_key_iv: "bc76d8d34aa6f5b4dccdf3780fee8bee"
server_key_version: "encryptionkey@0"
I20260927 16:28:39.652911 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:39.667776 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:39.670282 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:39.670486 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:39.671025 22939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:39.672008 22939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:39.672075 22939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.672139 22939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:39.672188 22939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:39.680613 22939 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:44169
I20260927 16:28:39.680645 23069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:44169 every 8 connection(s)
I20260927 16:28:39.681663 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:39.691815 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 22939
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.189@KRBTEST.COM
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.188@KRBTEST.COM
Sep 27 16:28:39 dist-test-slave-grzf krb5kdc[21787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526519, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:39.699878 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.683437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:55355 (local address 127.15.93.188:42007)
0927 16:28:39.683558 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.683562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.684427 (+ 865us) server_negotiation.cc:407] Connection header received
0927 16:28:39.685767 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.685770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.685821 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.685874 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.687127 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.687945 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689401 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689659 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689762 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.695213 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.695217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.695220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.695253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.697184 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.697930 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.697933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.697935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.698019 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698416 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.698419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.698421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.698637 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.698721 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.699595 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.699719 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:28:39.700901 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.683835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.15.93.131:55355)
0927 16:28:39.684283 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.684333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.685551 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.686057 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.686064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.686504 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.686973 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.686982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.688503 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.688506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.689298 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.689304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689879 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.689973 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.690631 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.690642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.695059 (+ 4417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.697315 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.697321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.697336 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.697813 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.698135 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.698139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.698311 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.698800 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.698805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.699408 (+ 603us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.700087 (+ 679us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.700212 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":119,"spinlock_wait_cycles":43648,"thread_start_us":135,"threads_started":1}
I20260927 16:28:39.701182 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.683825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:39359)
0927 16:28:39.684283 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.684333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.685550 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.686056 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.686062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.686448 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.686960 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.686973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.687828 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.687831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.689105 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.689114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689690 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.689883 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.690593 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.690615 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.693542 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.697315 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.697321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.697336 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.698045 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.698540 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.698545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.698618 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.699058 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.699064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.699288 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.699523 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.699859 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":73,"spinlock_wait_cycles":12288,"thread_start_us":108,"threads_started":1}
I20260927 16:28:39.701315 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.684920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:43905 (local address 127.15.93.189:37363)
0927 16:28:39.685043 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.685047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.685158 (+ 111us) server_negotiation.cc:407] Connection header received
0927 16:28:39.685766 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.685770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.685821 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.685874 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.687118 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.687945 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689103 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689427 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689520 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.693773 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.693780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.693786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.693816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.696373 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698147 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.698150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.698151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.698239 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.699699 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.699705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.699708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.699944 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.700038 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.701010 (+ 972us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.701140 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:39.701853 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.683402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:39359 (local address 127.15.93.190:44091)
0927 16:28:39.683600 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.683605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.684426 (+ 821us) server_negotiation.cc:407] Connection header received
0927 16:28:39.685746 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.685750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.685807 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.685882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.687101 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.687698 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689216 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689549 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689647 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.693711 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.693716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.693719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.693747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.697188 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698349 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.698355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.698359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.698440 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.698722 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.698726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.698729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.698922 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.699015 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.701557 (+ 2542us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.701685 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 16:28:39.700920 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.684809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.15.93.131:43905)
0927 16:28:39.685070 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.685083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.685551 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.686057 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.686064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.686505 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:39.686976 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.686982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.688047 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.688050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.688986 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.688996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.689689 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.689693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.689877 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.690703 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:39.690715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:39.693607 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:39.697491 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.697494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.697496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.697813 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.699466 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:39.699469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:39.699471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:39.699572 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:39.700282 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:39.700285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:39.700390 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.700500 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.700660 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":171,"spinlock_wait_cycles":5632,"thread_start_us":73,"threads_started":1}
I20260927 16:28:39.702347 23071 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44091
I20260927 16:28:39.702382 23070 heartbeater.cc:347] Connected to a master server at 127.15.93.188:42007
I20260927 16:28:39.702572 23070 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.702572 23071 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.703150 23071 heartbeater.cc:511] Master 127.15.93.190:44091 requested a full tablet report, sending...
I20260927 16:28:39.703960 23072 heartbeater.cc:347] Connected to a master server at 127.15.93.189:37363
I20260927 16:28:39.704038 23072 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:39.704202 23072 heartbeater.cc:511] Master 127.15.93.189:37363 requested a full tablet report, sending...
I20260927 16:28:39.704274 21832 ts_manager.cc:194] Re-registered known tserver with Master: 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:39.704921 22433 ts_manager.cc:194] Registered new tserver with Master: 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:39.704934 21832 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:39359
I20260927 16:28:39.703150 23070 heartbeater.cc:511] Master 127.15.93.188:42007 requested a full tablet report, sending...
I20260927 16:28:39.705634 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.694353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.15.93.189:37363)
0927 16:28:39.694607 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.694612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.694912 (+ 300us) server_negotiation.cc:407] Connection header received
0927 16:28:39.694954 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.694959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.695007 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.695085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.696136 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.696930 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.698308 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.698670 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.698757 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.701135 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.701138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.701141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.701161 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.702984 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.703522 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.703526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.703528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.703689 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.704006 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.704011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.704014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.704584 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.704724 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.705318 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.705433 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260927 16:28:39.706053 22616 ts_manager.cc:194] Registered new tserver with Master: 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:39.706238 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.693939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.15.93.190:44091)
0927 16:28:39.695108 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.695112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.695134 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:28:39.695168 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.695172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.695213 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.696127 (+ 914us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.696229 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.696978 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.698201 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.698532 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.698994 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.702295 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.702299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.702305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.702331 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.704148 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.704568 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.704571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.704573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.704659 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.704945 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.704949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.704951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.705171 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.705295 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.705950 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.706063 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":85,"threads_started":1}
I20260927 16:28:39.708845 21832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:39.712827 21832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:39.713765 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.694859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.15.93.188:42007)
0927 16:28:39.699231 (+ 4372us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.699235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.704515 (+ 5280us) server_negotiation.cc:407] Connection header received
0927 16:28:39.706658 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.706661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.706705 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.706759 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:39.707684 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.708178 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.709083 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.709416 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.709514 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.710707 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:39.710711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:39.710713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:39.710736 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:39.712244 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.712607 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.712611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.712612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.712709 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:39.712964 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:39.712967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:39.712969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:39.713180 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:39.713333 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.713481 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.713590 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4299,"thread_start_us":77,"threads_started":1}
I20260927 16:28:39.715600 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: NON_VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: true } } }
I20260927 16:28:39.715713 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.719012 22944 raft_consensus.cc:1050] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: attempting to promote NON_VOTER 103c04f460ca4c47a9bac35eaf77dc54 to VOTER
I20260927 16:28:39.719440 22944 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } }
I20260927 16:28:39.720651 22626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 LEARNER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:28:39.720741 22442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Refusing update from remote peer be0ed8ce3c0c44c0912336d08e3acc72: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 16:28:39.721042 23088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 16:28:39.721299 23088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 16:28:39.723258 22944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.723875 22442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.723784 22626 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.724483 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.724586 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.724777 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.724862 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.724965 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.725055 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:39.725956 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.726071 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:39.726365 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:39.726481 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:39.733011 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.727857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:45823 (local address 127.0.0.1:35150)
0927 16:28:39.728190 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.728206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.728282 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.729413 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.729418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.729441 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.729530 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.729536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.730809 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.730816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.731644 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.731652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.732645 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.732649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.732741 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.732775 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.732820 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.732887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260927 16:28:39.733255 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.728102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:43297 (local address 127.0.0.1:51880)
0927 16:28:39.728324 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.728337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.728424 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.729257 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.729260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.729273 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.729369 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.729374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.730748 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.730752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.731991 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.732001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.732847 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.732850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.732935 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.732958 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.733009 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.733084 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260927 16:28:39.733770 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.728456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.15.93.130:43297)
0927 16:28:39.728759 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.728770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.728798 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:28:39.728863 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.728867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.729090 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.729298 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.729460 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.730602 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.732091 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.732720 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.733438 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.733557 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.733596 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.733659 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260927 16:28:39.733774 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.728381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.15.93.129:45823)
0927 16:28:39.728845 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.728856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.728882 (+ 26us) server_negotiation.cc:407] Connection header received
0927 16:28:39.729012 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.729017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.729212 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.729690 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.729826 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.730701 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.731755 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.732509 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.733201 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.733496 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.733547 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.733645 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":94,"threads_started":1}
I20260927 16:28:39.735181 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.728831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44169 (local address 127.0.0.1:39162)
0927 16:28:39.729050 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.729064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.729140 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.730052 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.730055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.730073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.730172 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.730178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.731507 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.731510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.732679 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.732686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.734801 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.734804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.734897 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.734917 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.734996 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.735064 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":46,"thread_start_us":84,"threads_started":1}
I20260927 16:28:39.735244 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.729311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.15.93.131:44169)
0927 16:28:39.729635 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.729646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.729674 (+ 28us) server_negotiation.cc:407] Connection header received
0927 16:28:39.729739 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.729743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.729898 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.730061 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.730282 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.731339 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.732801 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.734677 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.734802 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.734928 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.735021 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.735105 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
I20260927 16:28:39.736846 22866 tablet_service.cc:1467] Processing CreateTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 (DEFAULT_TABLE table=test-table [id=72b78145a61540af977cbbbe6e290581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:39.736843 22729 tablet_service.cc:1467] Processing CreateTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 (DEFAULT_TABLE table=test-table [id=72b78145a61540af977cbbbe6e290581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:39.737586 23004 tablet_service.cc:1467] Processing CreateTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 (DEFAULT_TABLE table=test-table [id=72b78145a61540af977cbbbe6e290581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:39.737944 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06054788f29f4e2e993e53a8ae1540a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:39.737944 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06054788f29f4e2e993e53a8ae1540a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:39.738399 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06054788f29f4e2e993e53a8ae1540a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:39.744784 23104 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Bootstrap starting.
I20260927 16:28:39.745829 23105 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Bootstrap starting.
I20260927 16:28:39.747675 23106 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Bootstrap starting.
I20260927 16:28:39.747852 23104 tablet_bootstrap.cc:654] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:39.747852 23105 tablet_bootstrap.cc:654] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:39.748720 23104 log.cc:824] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:39.748800 23105 log.cc:824] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:39.749621 23106 tablet_bootstrap.cc:654] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:39.750387 23106 log.cc:824] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:39.750551 23105 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: No bootstrap required, opened a new log
I20260927 16:28:39.750758 23105 ts_tablet_manager.cc:1397] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:28:39.752274 23106 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: No bootstrap required, opened a new log
I20260927 16:28:39.752444 23106 ts_tablet_manager.cc:1397] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:28:39.752635 23104 tablet_bootstrap.cc:492] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: No bootstrap required, opened a new log
I20260927 16:28:39.752803 23104 ts_tablet_manager.cc:1397] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 16:28:39.755203 23105 raft_consensus.cc:348] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.755486 23105 raft_consensus.cc:374] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:39.755549 23105 raft_consensus.cc:729] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09bcd3a45014e7998b2d0ca225ffa69, State: Initialized, Role: FOLLOWER
I20260927 16:28:39.756002 23105 consensus_queue.cc:260] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.756264 23106 raft_consensus.cc:348] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.756500 23106 raft_consensus.cc:374] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:39.756611 23106 raft_consensus.cc:729] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72df7b22a1d94e02b9dca37e93fc30ab, State: Initialized, Role: FOLLOWER
I20260927 16:28:39.756682 23104 raft_consensus.cc:348] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.756894 23104 raft_consensus.cc:374] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:39.756940 23104 raft_consensus.cc:729] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18df5caabaed4b89903d368d5f8bdee3, State: Initialized, Role: FOLLOWER
I20260927 16:28:39.756992 23105 ts_tablet_manager.cc:1428] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 16:28:39.757038 23106 consensus_queue.cc:260] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.757210 22796 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.758185 23106 ts_tablet_manager.cc:1428] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:28:39.761368 22933 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.763530 23104 consensus_queue.cc:260] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.764183 23071 heartbeater.cc:503] Master 127.15.93.190:44091 was elected leader, sending a full tablet report...
I20260927 16:28:39.764420 23104 ts_tablet_manager.cc:1428] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
W20260927 16:28:39.770870 22798 tablet.cc:2369] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:39.914526 23111 raft_consensus.cc:482] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:39.914755 23111 raft_consensus.cc:504] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.916033 23111 leader_election.cc:288] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823), 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:39.919725 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.916312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:56281 (local address 127.15.93.129:45823)
0927 16:28:39.916474 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.916480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.916499 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:28:39.916593 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.916596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.916659 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.916748 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.917084 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.917666 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.918719 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.919102 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.919193 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.919257 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.919500 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.919586 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:28:39.919747 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.916240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:45823 (local address 127.15.93.130:56281)
0927 16:28:39.916373 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.916389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.916500 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.916808 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.916811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.916833 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.916972 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.916982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.917766 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.917769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.918590 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.918598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.919214 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.919221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.919361 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.919408 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.919483 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.919557 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":28}
I20260927 16:28:39.919783 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.916682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:47217 (local address 127.15.93.131:44169)
0927 16:28:39.916808 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.916812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.916824 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:39.916874 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.916877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.916922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.917008 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:39.917281 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.917798 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.918778 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.919181 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.919292 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.919349 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.919594 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.919670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:28:39.919755 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.916533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44169 (local address 127.15.93.130:47217)
0927 16:28:39.916682 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:39.916697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:39.916761 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:39.917020 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:39.917022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:39.917056 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:39.917166 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.917172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.917902 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.917904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.918632 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:39.918639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.919296 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.919300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:39.919464 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.919490 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:28:39.919565 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:28:39.919623 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1}
I20260927 16:28:39.920416 22749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06054788f29f4e2e993e53a8ae1540a7" candidate_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" is_pre_election: true
I20260927 16:28:39.920595 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06054788f29f4e2e993e53a8ae1540a7" candidate_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18df5caabaed4b89903d368d5f8bdee3" is_pre_election: true
I20260927 16:28:39.920699 22749 raft_consensus.cc:2454] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72df7b22a1d94e02b9dca37e93fc30ab in term 0.
I20260927 16:28:39.920899 23024 raft_consensus.cc:2454] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72df7b22a1d94e02b9dca37e93fc30ab in term 0.
I20260927 16:28:39.921191 22820 leader_election.cc:302] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72df7b22a1d94e02b9dca37e93fc30ab, b09bcd3a45014e7998b2d0ca225ffa69; no voters:
I20260927 16:28:39.921432 23111 raft_consensus.cc:2781] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:39.921504 23111 raft_consensus.cc:482] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:39.921532 23111 raft_consensus.cc:3037] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:39.922551 23111 raft_consensus.cc:504] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.922870 23111 leader_election.cc:288] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [CANDIDATE]: Term 1 election: Requested vote from peers b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823), 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169)
I20260927 16:28:39.923251 22749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06054788f29f4e2e993e53a8ae1540a7" candidate_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09bcd3a45014e7998b2d0ca225ffa69"
I20260927 16:28:39.923282 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06054788f29f4e2e993e53a8ae1540a7" candidate_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18df5caabaed4b89903d368d5f8bdee3"
I20260927 16:28:39.923403 22749 raft_consensus.cc:3037] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:39.923431 23024 raft_consensus.cc:3037] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:39.924618 22749 raft_consensus.cc:2454] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72df7b22a1d94e02b9dca37e93fc30ab in term 1.
I20260927 16:28:39.924618 23024 raft_consensus.cc:2454] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72df7b22a1d94e02b9dca37e93fc30ab in term 1.
I20260927 16:28:39.924898 22820 leader_election.cc:302] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72df7b22a1d94e02b9dca37e93fc30ab, b09bcd3a45014e7998b2d0ca225ffa69; no voters:
I20260927 16:28:39.925043 23111 raft_consensus.cc:2781] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:39.925274 23111 raft_consensus.cc:686] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 LEADER]: Becoming Leader. State: Replica: 72df7b22a1d94e02b9dca37e93fc30ab, State: Running, Role: LEADER
I20260927 16:28:39.925602 23111 consensus_queue.cc:237] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } }
I20260927 16:28:39.929881 21829 catalog_manager.cc:5495] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130). New cstate: current_term: 1 leader_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } health_report { overall_health: UNKNOWN } } }
W20260927 16:28:39.933351 23073 tablet.cc:2369] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:39.969146 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:39.965879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.15.93.130:43297)
0927 16:28:39.966020 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:39.966023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:39.966039 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:39.966177 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:39.966179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:39.966233 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:39.966331 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:39.966550 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.967031 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.967911 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:39.968126 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:39.968204 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:39.968453 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:39.968850 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:39.968935 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:39.968984 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:28:39.969033 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:28:39.974689 22749 raft_consensus.cc:1261] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Refusing update from remote peer 72df7b22a1d94e02b9dca37e93fc30ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:39.974689 23024 raft_consensus.cc:1261] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Refusing update from remote peer 72df7b22a1d94e02b9dca37e93fc30ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:39.975379 23113 consensus_queue.cc:1035] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:39.975570 23111 consensus_queue.cc:1035] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:39.982824 23116 mvcc.cc:205] Tried to move back new op lower bound from 7333996625809432576 to 7333996625617694720. Current Snapshot: MvccSnapshot[applied={T|T < 7333996625809432576}]
I20260927 16:28:39.987528 23118 mvcc.cc:205] Tried to move back new op lower bound from 7333996625809432576 to 7333996625617694720. Current Snapshot: MvccSnapshot[applied={T|T < 7333996625809432576}]
I20260927 16:28:39.995160 21829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:39.995327 21829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:40.000022 21829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be0ed8ce3c0c44c0912336d08e3acc72: Sending DeleteTablet for 3 replicas of tablet 06054788f29f4e2e993e53a8ae1540a7
I20260927 16:28:40.000854 22866 tablet_service.cc:1514] Processing DeleteTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:39 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:51880
I20260927 16:28:40.000885 22729 tablet_service.cc:1514] Processing DeleteTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:39 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35150
I20260927 16:28:40.000914 23004 tablet_service.cc:1514] Processing DeleteTablet for tablet 06054788f29f4e2e993e53a8ae1540a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:39 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:39162
I20260927 16:28:40.002892 23127 tablet_replica.cc:330] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: stopping tablet replica
I20260927 16:28:40.003017 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 21800
I20260927 16:28:40.003926 23127 raft_consensus.cc:2229] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:40.004259 23127 raft_consensus.cc:2258] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:40.004617 23126 tablet_replica.cc:330] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: stopping tablet replica
I20260927 16:28:40.004911 23126 raft_consensus.cc:2229] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:40.005199 23126 raft_consensus.cc:2258] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:40.005627 23125 tablet_replica.cc:330] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: stopping tablet replica
I20260927 16:28:40.005874 23125 raft_consensus.cc:2229] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:40.006212 23125 raft_consensus.cc:2258] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:40.007520 23127 ts_tablet_manager.cc:1905] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:40.007622 23126 ts_tablet_manager.cc:1905] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:40.010162 23126 ts_tablet_manager.cc:1918] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:40.010164 23127 ts_tablet_manager.cc:1918] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:40.010264 23127 log.cc:1196] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/06054788f29f4e2e993e53a8ae1540a7
I20260927 16:28:40.010264 23126 log.cc:1196] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/06054788f29f4e2e993e53a8ae1540a7
I20260927 16:28:40.010601 23127 ts_tablet_manager.cc:1939] T 06054788f29f4e2e993e53a8ae1540a7 P b09bcd3a45014e7998b2d0ca225ffa69: Deleting consensus metadata
I20260927 16:28:40.010613 23126 ts_tablet_manager.cc:1939] T 06054788f29f4e2e993e53a8ae1540a7 P 18df5caabaed4b89903d368d5f8bdee3: Deleting consensus metadata
I20260927 16:28:40.012570 23125 ts_tablet_manager.cc:1905] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 16:28:40.013130 22960 connection.cc:535] server connection from 127.0.0.1:39162 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 16:28:40.013746 22683 connection.cc:535] server connection from 127.0.0.1:35150 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:28:40.015184 23125 ts_tablet_manager.cc:1918] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:40.015277 23125 log.cc:1196] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/06054788f29f4e2e993e53a8ae1540a7
I20260927 16:28:40.015618 23125 ts_tablet_manager.cc:1939] T 06054788f29f4e2e993e53a8ae1540a7 P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting consensus metadata
W20260927 16:28:40.016623 22821 connection.cc:441] server connection from 127.0.0.1:51880 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51880 (request call id 1) could send its response
I20260927 16:28:40.020258 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.003795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.15.93.189:37363)
0927 16:28:40.003926 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:40.003933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:40.003948 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:40.003990 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:40.003995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:40.004048 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:40.004128 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:40.008726 (+ 4598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.009643 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.011315 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.011765 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.011861 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:40.014549 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:40.014556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:40.014562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:40.014593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:40.018396 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.018892 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.018896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.018898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.018982 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.019278 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.019282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.019284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.019524 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:40.019632 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:40.019957 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 16:28:40.020095 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:28:40.028386 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.018528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.15.93.189:37363)
0927 16:28:40.018659 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:40.018663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:40.018841 (+ 178us) server_negotiation.cc:407] Connection header received
0927 16:28:40.019022 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:40.019025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:40.019074 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:40.019125 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:40.020035 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.020873 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.022240 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.022565 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.022645 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:40.024812 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:40.024817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:40.024822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:40.024851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:40.026606 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.027022 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.027027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.027031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.027114 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.027441 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.027444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.027446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.027674 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:40.027778 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:40.028115 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 16:28:40.028229 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:28:40.028759 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.019000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.15.93.188:42007)
0927 16:28:40.019130 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:40.019134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:40.019656 (+ 522us) server_negotiation.cc:407] Connection header received
0927 16:28:40.019842 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:40.019845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:40.019898 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:40.019951 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:40.021177 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.021950 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.023138 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.023525 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.023606 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:40.025316 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:40.025320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:40.025323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:40.025347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:40.026922 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.027464 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.027467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.027468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.027541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.027846 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.027850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.027851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.028061 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:40.028163 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:40.028490 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 16:28:40.028618 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:28:40.030931 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.003081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.15.93.188:42007)
0927 16:28:40.003433 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:40.003437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:40.021059 (+ 17622us) server_negotiation.cc:407] Connection header received
0927 16:28:40.021226 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:40.021230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:40.021278 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:40.021342 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:40.022433 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.023207 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.024371 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:40.024715 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:40.024783 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:40.027703 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:40.027707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:40.027710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:40.027733 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:40.029343 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.029800 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.029804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.029806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.029880 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:40.030181 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:40.030184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:40.030185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:40.030333 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:40.030421 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:40.030665 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:28:40.030750 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260927 16:28:40.988597 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.988068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:46143)
0927 16:28:40.988341 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:40.988443 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
W20260927 16:28:40.989027 22933 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:40.989176 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.988592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:57193)
0927 16:28:40.988998 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:40.989082 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":72,"threads_started":1}
W20260927 16:28:40.989413 23071 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:40.989907 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:40.989409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:35179)
0927 16:28:40.989710 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:40.989803 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":60,"threads_started":1}
W20260927 16:28:40.990320 22796 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.499708 23138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be0ed8ce3c0c44c0912336d08e3acc72)
I20260927 16:28:41.499922 23138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } }
I20260927 16:28:41.500941 23138 leader_election.cc:288] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091), e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189:37363)
I20260927 16:28:41.501457 23139 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be0ed8ce3c0c44c0912336d08e3acc72)
I20260927 16:28:41.501699 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.501059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41792)
0927 16:28:41.501442 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.501555 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":57,"threads_started":1}
I20260927 16:28:41.501641 23139 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } }
W20260927 16:28:41.501878 22601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.502708 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.502471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41806)
0927 16:28:41.502551 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.502631 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 16:28:41.502944 22601 leader_election.cc:334] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091): Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.503199 23139 leader_election.cc:288] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091), 103c04f460ca4c47a9bac35eaf77dc54 (127.15.93.188:42007)
I20260927 16:28:41.503356 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.502737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41820)
0927 16:28:41.503192 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.503262 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":60,"threads_started":1}
W20260927 16:28:41.503536 22418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.504400 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.504186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41832)
0927 16:28:41.504292 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.504341 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 16:28:41.504494 22418 leader_election.cc:334] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091): Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.505734 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.501224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.15.93.189:37363)
0927 16:28:41.501548 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.501553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.501569 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:41.501711 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.501717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.501776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.501865 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.502222 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.503070 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.504782 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.505285 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.505363 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.505407 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.505605 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.505656 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":249,"thread_start_us":55,"threads_started":1}
I20260927 16:28:41.505780 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.501177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37363 (local address 127.0.0.1:35672)
0927 16:28:41.501380 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.501396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.501500 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.501922 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.501925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.501945 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.502088 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.502095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.503169 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.503173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.504659 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.504669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.505396 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.505400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.505489 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.505514 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.505596 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.505660 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260927 16:28:41.506268 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103c04f460ca4c47a9bac35eaf77dc54" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e47accc20b3a40638b37bfcfc2cf1493" is_pre_election: true
I20260927 16:28:41.506450 22442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103c04f460ca4c47a9bac35eaf77dc54 in term 1.
I20260927 16:28:41.506783 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 103c04f460ca4c47a9bac35eaf77dc54, e47accc20b3a40638b37bfcfc2cf1493; no voters: be0ed8ce3c0c44c0912336d08e3acc72
I20260927 16:28:41.506991 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 16:28:41.507042 23138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Starting leader election (detected failure of leader be0ed8ce3c0c44c0912336d08e3acc72)
I20260927 16:28:41.507078 23138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:28:41.507187 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.502795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:42007 (local address 127.0.0.1:50602)
0927 16:28:41.503432 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.503447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.503564 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.503940 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.503943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.503961 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.504040 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.504045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.504917 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.504920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.506003 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.506011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.506789 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.506792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.506878 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.506898 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.506990 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.507057 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260927 16:28:41.507207 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.503111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.15.93.188:42007)
0927 16:28:41.503429 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.503439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.503522 (+ 83us) server_negotiation.cc:407] Connection header received
0927 16:28:41.503707 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.503710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.503756 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.503849 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.504145 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.504814 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.506115 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.506676 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.506790 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.506841 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.507003 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.507063 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260927 16:28:41.507643 22626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e47accc20b3a40638b37bfcfc2cf1493" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "103c04f460ca4c47a9bac35eaf77dc54" is_pre_election: true
I20260927 16:28:41.508294 23138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } }
I20260927 16:28:41.508525 22626 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e47accc20b3a40638b37bfcfc2cf1493 in current term 2: Already voted for candidate 103c04f460ca4c47a9bac35eaf77dc54 in this term.
I20260927 16:28:41.508671 23138 leader_election.cc:288] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 election: Requested vote from peers be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091), e47accc20b3a40638b37bfcfc2cf1493 (127.15.93.189:37363)
I20260927 16:28:41.508953 22419 leader_election.cc:302] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e47accc20b3a40638b37bfcfc2cf1493; no voters: 103c04f460ca4c47a9bac35eaf77dc54, be0ed8ce3c0c44c0912336d08e3acc72
I20260927 16:28:41.508970 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103c04f460ca4c47a9bac35eaf77dc54" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e47accc20b3a40638b37bfcfc2cf1493"
I20260927 16:28:41.509124 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:28:41.509433 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.508765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41836)
0927 16:28:41.509344 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.509386 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
I20260927 16:28:41.510022 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.509850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41842)
0927 16:28:41.509939 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.509979 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:28:41.510156 22601 leader_election.cc:334] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091): Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
I20260927 16:28:41.510318 22442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103c04f460ca4c47a9bac35eaf77dc54 in term 2.
I20260927 16:28:41.510425 23139 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:28:41.510594 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 103c04f460ca4c47a9bac35eaf77dc54, e47accc20b3a40638b37bfcfc2cf1493; no voters: be0ed8ce3c0c44c0912336d08e3acc72
I20260927 16:28:41.510807 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:28:41.511003 23138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [term 2 LEADER]: Becoming Leader. State: Replica: 103c04f460ca4c47a9bac35eaf77dc54, State: Running, Role: LEADER
I20260927 16:28:41.511231 23138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } }
I20260927 16:28:41.512444 23148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103c04f460ca4c47a9bac35eaf77dc54. Latest consensus state: current_term: 2 leader_uuid: "103c04f460ca4c47a9bac35eaf77dc54" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:41.512537 23148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:41.512809 23150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:41.514832 23150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:41.515246 23150 catalog_manager.cc:1259] Loaded cluster ID: 36bf8cf5459a4de38a75de51df30a930
I20260927 16:28:41.515311 23150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:41.515899 23150 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:41.516296 23150 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Loaded TSK: 0
I20260927 16:28:41.516989 23150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:28:41.535269 22615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:41.538223 22615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:41.540925 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.540689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41864)
0927 16:28:41.540808 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.540860 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 16:28:41.540944 22442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [term 2 FOLLOWER]: Refusing update from remote peer 103c04f460ca4c47a9bac35eaf77dc54: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 16:28:41.541464 23148 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 16:28:41.541603 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.541388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.0.0.1:41872)
0927 16:28:41.541495 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.541543 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:28:41.541680 22601 consensus_peers.cc:487] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 -> Peer be0ed8ce3c0c44c0912336d08e3acc72 (127.15.93.190:44091): Couldn't send request to peer be0ed8ce3c0c44c0912336d08e3acc72. Status: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 16:28:41.542811 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103c04f460ca4c47a9bac35eaf77dc54. Latest consensus state: current_term: 2 leader_uuid: "103c04f460ca4c47a9bac35eaf77dc54" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:41.542924 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:41.544186 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "103c04f460ca4c47a9bac35eaf77dc54" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:41.544282 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:41.544585 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "103c04f460ca4c47a9bac35eaf77dc54" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "be0ed8ce3c0c44c0912336d08e3acc72" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44091 } } peers { permanent_uuid: "e47accc20b3a40638b37bfcfc2cf1493" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37363 } attrs { promote: false } } peers { permanent_uuid: "103c04f460ca4c47a9bac35eaf77dc54" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 42007 } attrs { promote: false } } }
I20260927 16:28:41.544693 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47accc20b3a40638b37bfcfc2cf1493 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:28:41.544847 23152 mvcc.cc:205] Tried to move back new op lower bound from 7333996632227565568 to 7333996632112205824. Current Snapshot: MvccSnapshot[applied={T|T < 7333996632227565568}]
I20260927 16:28:41.554095 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.550082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:43297 (local address 127.0.0.1:51900)
0927 16:28:41.550317 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.550331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.550414 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.550820 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.550823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.550838 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.550918 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.550924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.551785 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.551789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.552785 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.552793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.553708 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.553712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.553798 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.553823 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.553903 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.553965 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260927 16:28:41.554131 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.550407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.15.93.130:43297)
0927 16:28:41.550626 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.550629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.550641 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:41.550682 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.550684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.550720 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.550814 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.551036 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.551669 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.552894 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.553548 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.553691 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.553754 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.553926 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.553999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260927 16:28:41.554670 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.549554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44169 (local address 127.0.0.1:39174)
0927 16:28:41.549676 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.549690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.549771 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.550286 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.550292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.550308 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.550413 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.550421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.551519 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.551522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.552340 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.552348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.554288 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.554293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.554387 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.554404 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.554487 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.554554 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 16:28:41.554848 22866 tablet_service.cc:1467] Processing CreateTablet for tablet 1e07a5087cc643fd84206d882aabf75a (DEFAULT_TABLE table=test-table [id=9d047d3a5e464355bee35f482d830fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:41.554950 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.549830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:45823 (local address 127.0.0.1:35160)
0927 16:28:41.550156 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.550170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.550255 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.551255 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.551258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.551274 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.551348 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.551352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.552458 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.552463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.553392 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.553402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.554557 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.554561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.554650 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.554670 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.554767 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.554842 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260927 16:28:41.555133 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e07a5087cc643fd84206d882aabf75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:41.555243 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.549701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.15.93.131:44169)
0927 16:28:41.549924 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.549929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.549944 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:41.549999 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.550002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.550155 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.550525 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.550641 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.551423 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.553537 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.554151 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.554916 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.554983 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.555018 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.555096 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":60,"threads_started":1}
I20260927 16:28:41.555335 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.550079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.15.93.129:45823)
0927 16:28:41.551028 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.551032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.551046 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:41.551093 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.551095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.551139 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.551473 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.551590 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.552338 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.553563 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.554388 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.554546 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.554602 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.554904 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.555173 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":81,"threads_started":1}
I20260927 16:28:41.555809 23004 tablet_service.cc:1467] Processing CreateTablet for tablet 1e07a5087cc643fd84206d882aabf75a (DEFAULT_TABLE table=test-table [id=9d047d3a5e464355bee35f482d830fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:41.556061 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e07a5087cc643fd84206d882aabf75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:41.557637 23161 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Bootstrap starting.
I20260927 16:28:41.558465 23162 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Bootstrap starting.
I20260927 16:28:41.558636 23161 tablet_bootstrap.cc:654] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:41.558650 22729 tablet_service.cc:1467] Processing CreateTablet for tablet 1e07a5087cc643fd84206d882aabf75a (DEFAULT_TABLE table=test-table [id=9d047d3a5e464355bee35f482d830fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:41.558900 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e07a5087cc643fd84206d882aabf75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:41.559185 23162 tablet_bootstrap.cc:654] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:41.560101 23161 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: No bootstrap required, opened a new log
I20260927 16:28:41.560249 23161 ts_tablet_manager.cc:1397] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:41.560643 23161 raft_consensus.cc:348] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.560757 23161 raft_consensus.cc:374] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:41.560804 23161 raft_consensus.cc:729] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72df7b22a1d94e02b9dca37e93fc30ab, State: Initialized, Role: FOLLOWER
I20260927 16:28:41.560935 23161 consensus_queue.cc:260] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.561282 23166 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69: Bootstrap starting.
I20260927 16:28:41.561316 23161 ts_tablet_manager.cc:1428] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:41.562153 23166 tablet_bootstrap.cc:654] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:41.562187 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.561727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:53803)
0927 16:28:41.562040 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.562096 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":84,"threads_started":1}
I20260927 16:28:41.562285 23162 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: No bootstrap required, opened a new log
I20260927 16:28:41.562359 23162 ts_tablet_manager.cc:1397] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 16:28:41.562788 23162 raft_consensus.cc:348] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.562893 23162 raft_consensus.cc:374] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:41.562922 23162 raft_consensus.cc:729] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18df5caabaed4b89903d368d5f8bdee3, State: Initialized, Role: FOLLOWER
I20260927 16:28:41.563053 23162 consensus_queue.cc:260] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.563338 23166 tablet_bootstrap.cc:492] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69: No bootstrap required, opened a new log
I20260927 16:28:41.563410 23162 ts_tablet_manager.cc:1428] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:41.563441 23166 ts_tablet_manager.cc:1397] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 16:28:41.563875 23166 raft_consensus.cc:348] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.563992 23166 raft_consensus.cc:374] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:41.564026 23166 raft_consensus.cc:729] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09bcd3a45014e7998b2d0ca225ffa69, State: Initialized, Role: FOLLOWER
I20260927 16:28:41.564167 23166 consensus_queue.cc:260] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.564381 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.563880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:40547)
0927 16:28:41.564233 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.564296 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":86,"threads_started":1}
I20260927 16:28:41.564481 23166 ts_tablet_manager.cc:1428] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 16:28:41.564671 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.564403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:44045)
0927 16:28:41.564537 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.564604 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 16:28:41.565511 22932 heartbeater.cc:503] Master 127.15.93.188:42007 was elected leader, sending a full tablet report...
I20260927 16:28:41.565740 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.565364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:47645)
0927 16:28:41.565567 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.565650 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260927 16:28:41.567054 22795 heartbeater.cc:503] Master 127.15.93.188:42007 was elected leader, sending a full tablet report...
I20260927 16:28:41.567873 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.567659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:48359)
0927 16:28:41.567768 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.567813 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 16:28:41.568260 23070 heartbeater.cc:503] Master 127.15.93.188:42007 was elected leader, sending a full tablet report...
I20260927 16:28:41.568626 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.566318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:41175)
0927 16:28:41.568509 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.568562 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2114}
I20260927 16:28:41.670140 23171 raft_consensus.cc:482] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:41.670302 23171 raft_consensus.cc:504] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.671379 23171 leader_election.cc:288] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169), 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:41.675031 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.671747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:50281 (local address 127.15.93.130:43297)
0927 16:28:41.671866 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.671870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.671886 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:41.672056 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.672058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.672100 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.672186 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.672460 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.672981 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674029 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.674407 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674560 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.674628 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.674823 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.674895 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:28:41.675031 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.671616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:43297 (local address 127.15.93.129:50281)
0927 16:28:41.671797 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.671813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.671916 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.672197 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.672200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.672218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.672344 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.672349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.673095 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.673100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.673898 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.673906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.674558 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.674676 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.674720 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.674806 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.674874 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":3}
I20260927 16:28:41.675063 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.671715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:55225 (local address 127.15.93.131:44169)
0927 16:28:41.671841 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.671846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.672077 (+ 231us) server_negotiation.cc:407] Connection header received
0927 16:28:41.672243 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.672246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.672295 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.672380 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:41.672631 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.673110 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674083 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.674487 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674617 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.674676 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.674875 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.674942 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:28:41.675288 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.671618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:44169 (local address 127.15.93.129:55225)
0927 16:28:41.671978 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.671989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:41.672136 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:41.672391 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:41.672394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:41.672409 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:41.672519 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.672524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.673223 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.673228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.673959 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:41.673967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.674610 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.674613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:41.674753 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.674770 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:28:41.674859 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:28:41.674927 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":21,"thread_start_us":123,"threads_started":1}
I20260927 16:28:41.675565 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e07a5087cc643fd84206d882aabf75a" candidate_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" is_pre_election: true
I20260927 16:28:41.675820 22886 raft_consensus.cc:2454] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09bcd3a45014e7998b2d0ca225ffa69 in term 0.
I20260927 16:28:41.675827 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e07a5087cc643fd84206d882aabf75a" candidate_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18df5caabaed4b89903d368d5f8bdee3" is_pre_election: true
I20260927 16:28:41.675963 23024 raft_consensus.cc:2454] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09bcd3a45014e7998b2d0ca225ffa69 in term 0.
I20260927 16:28:41.676291 22685 leader_election.cc:302] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18df5caabaed4b89903d368d5f8bdee3, 72df7b22a1d94e02b9dca37e93fc30ab, b09bcd3a45014e7998b2d0ca225ffa69; no voters:
I20260927 16:28:41.676570 23171 raft_consensus.cc:2781] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:41.676641 23171 raft_consensus.cc:482] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:41.676699 23171 raft_consensus.cc:3037] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:41.677682 23171 raft_consensus.cc:504] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.678016 23171 leader_election.cc:288] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [CANDIDATE]: Term 1 election: Requested vote from peers 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169), 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297)
I20260927 16:28:41.678390 23024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e07a5087cc643fd84206d882aabf75a" candidate_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18df5caabaed4b89903d368d5f8bdee3"
I20260927 16:28:41.678388 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e07a5087cc643fd84206d882aabf75a" candidate_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72df7b22a1d94e02b9dca37e93fc30ab"
I20260927 16:28:41.678511 22886 raft_consensus.cc:3037] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:41.678563 23024 raft_consensus.cc:3037] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:41.679679 23024 raft_consensus.cc:2454] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09bcd3a45014e7998b2d0ca225ffa69 in term 1.
I20260927 16:28:41.679729 22886 raft_consensus.cc:2454] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09bcd3a45014e7998b2d0ca225ffa69 in term 1.
I20260927 16:28:41.679955 22683 leader_election.cc:302] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18df5caabaed4b89903d368d5f8bdee3, b09bcd3a45014e7998b2d0ca225ffa69; no voters:
I20260927 16:28:41.680128 23171 raft_consensus.cc:2781] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:41.680356 23171 raft_consensus.cc:686] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [term 1 LEADER]: Becoming Leader. State: Replica: b09bcd3a45014e7998b2d0ca225ffa69, State: Running, Role: LEADER
I20260927 16:28:41.680631 23171 consensus_queue.cc:237] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } }
I20260927 16:28:41.683132 22614 catalog_manager.cc:5495] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b09bcd3a45014e7998b2d0ca225ffa69" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 45823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 } health_report { overall_health: UNKNOWN } } }
I20260927 16:28:41.685103 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.684833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:42149)
0927 16:28:41.684963 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.685027 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 16:28:41.704457 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.701035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.15.93.129:45823)
0927 16:28:41.701152 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:41.701156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:41.701169 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:28:41.701312 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:41.701314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:41.701363 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:41.701447 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:41.701659 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.702236 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.703007 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:41.703225 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:41.703317 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:41.703611 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:41.704099 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:41.704189 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:41.704245 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:28:41.704293 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:41.707005 23024 raft_consensus.cc:1261] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Refusing update from remote peer b09bcd3a45014e7998b2d0ca225ffa69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:41.707262 22886 raft_consensus.cc:1261] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Refusing update from remote peer b09bcd3a45014e7998b2d0ca225ffa69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:41.707566 23174 consensus_queue.cc:1035] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18df5caabaed4b89903d368d5f8bdee3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 44169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:41.707845 23171 consensus_queue.cc:1035] T 1e07a5087cc643fd84206d882aabf75a P b09bcd3a45014e7998b2d0ca225ffa69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72df7b22a1d94e02b9dca37e93fc30ab" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:41.709281 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.708975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:39281)
0927 16:28:41.709151 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.709211 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 16:28:41.709623 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.709299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:60745)
0927 16:28:41.709450 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.709499 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 16:28:41.712636 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.712374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.130:42531)
0927 16:28:41.712524 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.712573 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 16:28:41.713951 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.713688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.129:52455)
0927 16:28:41.713827 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.713879 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 16:28:41.714733 23178 mvcc.cc:205] Tried to move back new op lower bound from 7333996632907141120 to 7333996632806330368. Current Snapshot: MvccSnapshot[applied={T|T < 7333996632907141120}]
I20260927 16:28:41.715561 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.715272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44091 (local address 127.15.93.131:53351)
0927 16:28:41.715417 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.715477 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:44091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 16:28:41.722810 22615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:41.722936 22615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:41.726647 22615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 103c04f460ca4c47a9bac35eaf77dc54: Sending DeleteTablet for 3 replicas of tablet 1e07a5087cc643fd84206d882aabf75a
I20260927 16:28:41.727267 23004 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e07a5087cc643fd84206d882aabf75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:41 UTC) from {username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:39174
I20260927 16:28:41.727424 22866 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e07a5087cc643fd84206d882aabf75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:41 UTC) from {username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:51900
I20260927 16:28:41.727483 22729 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e07a5087cc643fd84206d882aabf75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:41 UTC) from {username='oryx', principal='oryx/127.15.93.188@KRBTEST.COM'} at 127.0.0.1:35160
I20260927 16:28:41.727928 23188 tablet_replica.cc:330] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: stopping tablet replica
I20260927 16:28:41.728065 23188 raft_consensus.cc:2229] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:41.728076 23187 tablet_replica.cc:330] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: stopping tablet replica
I20260927 16:28:41.728204 23188 raft_consensus.cc:2258] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:41.728266 23187 raft_consensus.cc:2229] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:41.728320 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22578
I20260927 16:28:41.728433 23187 raft_consensus.cc:2258] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:41.730162 23187 ts_tablet_manager.cc:1905] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:41.731734 23188 ts_tablet_manager.cc:1905] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:41.733502 23187 ts_tablet_manager.cc:1918] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:41.733587 23187 log.cc:1196] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/1e07a5087cc643fd84206d882aabf75a
I20260927 16:28:41.733896 23187 ts_tablet_manager.cc:1939] T 1e07a5087cc643fd84206d882aabf75a P 18df5caabaed4b89903d368d5f8bdee3: Deleting consensus metadata
I20260927 16:28:41.733954 23188 ts_tablet_manager.cc:1918] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:41.734037 23188 log.cc:1196] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/1e07a5087cc643fd84206d882aabf75a
I20260927 16:28:41.734359 23188 ts_tablet_manager.cc:1939] T 1e07a5087cc643fd84206d882aabf75a P 72df7b22a1d94e02b9dca37e93fc30ab: Deleting consensus metadata
I20260927 16:28:41.734668 22601 catalog_manager.cc:4841] TS 18df5caabaed4b89903d368d5f8bdee3 (127.15.93.131:44169): tablet 1e07a5087cc643fd84206d882aabf75a (table test-table [id=9d047d3a5e464355bee35f482d830fa7]) successfully deleted
I20260927 16:28:41.735018 22603 catalog_manager.cc:4841] TS 72df7b22a1d94e02b9dca37e93fc30ab (127.15.93.130:43297): tablet 1e07a5087cc643fd84206d882aabf75a (table test-table [id=9d047d3a5e464355bee35f482d830fa7]) successfully deleted
I20260927 16:28:41.736393 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22803
I20260927 16:28:41.736632 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:41.736381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:45823 (local address 127.0.0.1:35180)
0927 16:28:41.736480 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:41.736535 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:45823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 16:28:41.737398 22601 catalog_manager.cc:4584] TS b09bcd3a45014e7998b2d0ca225ffa69 (127.15.93.129:45823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e07a5087cc643fd84206d882aabf75a: Network error: Client connection negotiation failed: client connection to 127.15.93.129:45823: connect: Connection refused (error 111)
I20260927 16:28:41.743170 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22939
I20260927 16:28:41.748835 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22296
I20260927 16:28:41.754508 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 22398
2026-09-27T16:28:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5324 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:41 dist-test-slave-grzf krb5kdc[23192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:41 dist-test-slave-grzf krb5kdc[23192](info): set up 2 sockets
Sep 27 16:28:41 dist-test-slave-grzf krb5kdc[23192](info): commencing operation
krb5kdc: starting...
W20260927 16:28:42.788049 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:42 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab.
Entry for principal oryx/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab.
I20260927 16:28:42.901615 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43173
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab
--principal=oryx/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:43.006615 23205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:43.006865 23205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:43.006945 23205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:43.010879 23205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:43.010938 23205 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 16:28:43.010955 23205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:43.010974 23205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:43.010990 23205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:43.011039 23205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:43.016275 23205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:43173
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.190.keytab
--principal=oryx/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:43.017282 23205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:43.018268 23205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:43.026208 23210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.026237 23211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.026207 23213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.026513 23205 server_base.cc:1034] running on GCE node
I20260927 16:28:43.027118 23205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:43.028188 23205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:43.029371 23205 hybrid_clock.cc:648] HybridClock initialized: now 1790526523029348 us; error 43 us; skew 500 ppm
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:43.031957 23205 init.cc:377] Logged in from keytab as oryx/127.15.93.190@KRBTEST.COM (short username oryx)
I20260927 16:28:43.032792 23205 webserver.cc:460] Webserver started at http://127.15.93.190:42903/ using document root <none> and password file <none>
I20260927 16:28:43.033315 23205 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:43.033403 23205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:43.033627 23205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:43.035589 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "0e2907a06107478e9bdea634a5dbad4e"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "35803d641e631615c96333c93d3a1421"
server_key_iv: "65bae36908e89905d7f2ac2b9d913b34"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.036123 23205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "0e2907a06107478e9bdea634a5dbad4e"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "35803d641e631615c96333c93d3a1421"
server_key_iv: "65bae36908e89905d7f2ac2b9d913b34"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.039501 23205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:43.041654 23220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.042737 23205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:43.042867 23205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0e2907a06107478e9bdea634a5dbad4e"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "35803d641e631615c96333c93d3a1421"
server_key_iv: "65bae36908e89905d7f2ac2b9d913b34"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.042968 23205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:43.054987 23205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:43.057483 23205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:43.057677 23205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:43.065346 23205 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:43173
I20260927 16:28:43.065387 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:43173 every 8 connection(s)
I20260927 16:28:43.066260 23205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:43.066933 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23205
I20260927 16:28:43.067054 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:43.067354 15734 external_mini_cluster.cc:1428] Setting key 1faa174e34493c3fe34919e317103e0b
I20260927 16:28:43.069444 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:43.074837 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Bootstrap starting.
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:43.077467 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:43.078655 23283 log.cc:824] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:43.080904 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: No bootstrap required, opened a new log
I20260927 16:28:43.081297 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.068759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.15.93.190:43173)
0927 16:28:43.069179 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.069193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.069227 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:28:43.069898 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.069916 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.070233 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.070559 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.071353 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.072391 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.073243 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.073701 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.073868 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.076610 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.076616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.076630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.076656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.078324 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.078807 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.078814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.078820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.078942 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.079337 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.079344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.079346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.079815 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.079954 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.080261 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.080551 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260927 16:28:43.084272 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } }
I20260927 16:28:43.084474 23283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:43.084544 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2907a06107478e9bdea634a5dbad4e, State: Initialized, Role: FOLLOWER
I20260927 16:28:43.084942 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } }
I20260927 16:28:43.085080 23283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:43.085142 23283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:43.085263 23283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:43.086161 23283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } }
I20260927 16:28:43.086493 23283 leader_election.cc:302] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e2907a06107478e9bdea634a5dbad4e; no voters:
I20260927 16:28:43.086815 23283 leader_election.cc:288] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:43.086921 23288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:43.087165 23288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [term 1 LEADER]: Becoming Leader. State: Replica: 0e2907a06107478e9bdea634a5dbad4e, State: Running, Role: LEADER
I20260927 16:28:43.087554 23288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } }
I20260927 16:28:43.088063 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:43.089087 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e2907a06107478e9bdea634a5dbad4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } } }
I20260927 16:28:43.089217 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:43.089586 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e2907a06107478e9bdea634a5dbad4e. Latest consensus state: current_term: 1 leader_uuid: "0e2907a06107478e9bdea634a5dbad4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e2907a06107478e9bdea634a5dbad4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43173 } } }
I20260927 16:28:43.089730 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e [sys.catalog]: This master's current role is: LEADER
W20260927 16:28:43.092712 23303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:28:43.092774 23303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:28:43.092852 23297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:43.093528 23297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:43.099182 23297 catalog_manager.cc:1347] Generated new cluster ID: 9ce72d34efd249cea5d74a4df765e0df
I20260927 16:28:43.099241 23297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:43.109350 23297 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:43.110468 23297 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:43.114346 23297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Generated new TSK 0
I20260927 16:28:43.115031 23297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
I20260927 16:28:43.142467 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43173
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:43.249375 23309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:43.249614 23309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:43.249694 23309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:43.253656 23309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:43.253712 23309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:43.253750 23309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:43.253782 23309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:43.258868 23309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:43.259935 23309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:43.260907 23309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:43.270110 23315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.270200 23317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.270179 23309 server_base.cc:1034] running on GCE node
W20260927 16:28:43.270174 23314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.270746 23309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:43.271333 23309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:43.272532 23309 hybrid_clock.cc:648] HybridClock initialized: now 1790526523272519 us; error 28 us; skew 500 ppm
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:43.275178 23309 init.cc:377] Logged in from keytab as oryx/127.15.93.129@KRBTEST.COM (short username oryx)
I20260927 16:28:43.276036 23309 webserver.cc:460] Webserver started at http://127.15.93.129:37539/ using document root <none> and password file <none>
I20260927 16:28:43.276580 23309 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:43.276686 23309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:43.276912 23309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:43.278867 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "d8b1aac900c086cec90fbe29e716acbd"
server_key_iv: "c931e11c5cc40b9c0bf633594b2b6680"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.279474 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "d8b1aac900c086cec90fbe29e716acbd"
server_key_iv: "c931e11c5cc40b9c0bf633594b2b6680"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.282923 23309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:28:43.285277 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.286432 23309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:28:43.286563 23309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "d8b1aac900c086cec90fbe29e716acbd"
server_key_iv: "c931e11c5cc40b9c0bf633594b2b6680"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.286684 23309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:43.304605 23309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:43.307592 23309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:43.307791 23309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:43.309238 23309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:43.309309 23309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.309422 23309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:43.309477 23309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.319018 23309 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:32865
I20260927 16:28:43.319128 23439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:32865 every 8 connection(s)
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:43.320122 23309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:43.323664 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.310091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:32945 (local address 127.15.93.190:43173)
0927 16:28:43.310233 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.310238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.311100 (+ 862us) server_negotiation.cc:407] Connection header received
0927 16:28:43.312319 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.312322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.312383 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.312469 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.313957 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.314886 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.316764 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.317177 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.317286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.320026 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.320029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.320031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.320056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.321530 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.322089 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.322092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.322094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.322183 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.322531 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.322534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.322536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.322735 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.322839 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.323331 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.323483 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:28:43.324426 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.310503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.129:32945)
0927 16:28:43.310938 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.310986 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.312056 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.312658 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.312669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.313133 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.313809 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.313829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.315054 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.315061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.316618 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.316630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.317322 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.317326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.317510 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.318137 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.318153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.319855 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.321695 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.321701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.321715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.321974 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.322284 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.322287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.322289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.322429 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.322850 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.322858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.323091 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.323275 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.323605 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260927 16:28:43.325788 23440 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43173
I20260927 16:28:43.325939 23440 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:43.326438 23440 heartbeater.cc:511] Master 127.15.93.190:43173 requested a full tablet report, sending...
I20260927 16:28:43.328022 23236 ts_manager.cc:194] Registered new tserver with Master: 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865)
I20260927 16:28:43.328179 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23309
I20260927 16:28:43.328297 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:43.328591 15734 external_mini_cluster.cc:1428] Setting key f29b80e32aeaace4e3259403cd3c8697
I20260927 16:28:43.329561 23236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:32945
I20260927 16:28:43.336520 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.328014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:45861 (local address 127.15.93.190:43173)
0927 16:28:43.328127 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.328131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.329142 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 16:28:43.329300 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.329303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.329345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.329426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.329937 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.330588 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.331745 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.332052 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.332112 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.333275 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.333278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.333281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.333305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.334759 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.335289 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.335292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.335293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.335368 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.335738 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.335740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.335741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.335899 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.335973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.336314 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.336406 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:28:43.336671 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.327890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.129:45861)
0927 16:28:43.329063 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.329077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.329172 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.329460 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.329463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.329688 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.329805 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.329812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.330759 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.330762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.331602 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.331612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.332233 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.332235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.332299 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.332809 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.332821 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.333140 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.334894 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.334900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.334905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.335181 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.335520 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.335526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.335530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.335634 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.335994 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.335999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.336118 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.336267 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.336439 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
I20260927 16:28:43.356357 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43173
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:43.464305 23449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:43.464605 23449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:43.464686 23449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:43.468624 23449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:43.468711 23449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:43.468782 23449 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:43.468863 23449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:43.473891 23449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:43.474925 23449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:43.475935 23449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:43.484961 23454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.485020 23457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.484965 23455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.485196 23449 server_base.cc:1034] running on GCE node
I20260927 16:28:43.485671 23449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:43.486277 23449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:43.487470 23449 hybrid_clock.cc:648] HybridClock initialized: now 1790526523487451 us; error 28 us; skew 500 ppm
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:43.490273 23449 init.cc:377] Logged in from keytab as oryx/127.15.93.130@KRBTEST.COM (short username oryx)
I20260927 16:28:43.491175 23449 webserver.cc:460] Webserver started at http://127.15.93.130:36815/ using document root <none> and password file <none>
I20260927 16:28:43.491787 23449 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:43.491874 23449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:43.492096 23449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:43.494131 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "4d4d414f14454310828190f716607b99"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "0b385b874f538b3f36de894ec4b57cfb"
server_key_iv: "98b9aaa4ef73cc4f1c2959796816d98b"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.494712 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "4d4d414f14454310828190f716607b99"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "0b385b874f538b3f36de894ec4b57cfb"
server_key_iv: "98b9aaa4ef73cc4f1c2959796816d98b"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.498291 23449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:28:43.500823 23464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.502032 23449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:43.502173 23449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "4d4d414f14454310828190f716607b99"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "0b385b874f538b3f36de894ec4b57cfb"
server_key_iv: "98b9aaa4ef73cc4f1c2959796816d98b"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.502280 23449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:43.535606 23449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:43.538395 23449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:43.538599 23449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:43.540150 23449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:43.540221 23449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.540307 23449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:43.540359 23449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:43.551465 23449 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:40249
I20260927 16:28:43.551709 23579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:40249 every 8 connection(s)
I20260927 16:28:43.552450 23449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:43.554080 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.540867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:50527 (local address 127.15.93.190:43173)
0927 16:28:43.541014 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.541019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.541936 (+ 917us) server_negotiation.cc:407] Connection header received
0927 16:28:43.543314 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.543321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.543411 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.543508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.544640 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.545311 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.546664 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.546957 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.547065 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.550561 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.550564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.550567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.550597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.552149 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.552682 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.552687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.552689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.552771 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.553162 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.553165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.553168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.553356 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.553432 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.553872 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.553964 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:28:43.554843 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.541274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.130:50527)
0927 16:28:43.541750 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.541812 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.543019 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.543699 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.543709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.544032 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.544489 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.544501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.545433 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.545443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.546543 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.546552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.547111 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.547123 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.547326 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.547964 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.547994 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.550401 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.552258 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.552264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.552278 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.552573 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.552865 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.552868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.552871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.553035 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.553456 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.553461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.553675 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.553829 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.554054 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260927 16:28:43.555941 23580 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43173
I20260927 16:28:43.556093 23580 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:43.556598 23580 heartbeater.cc:511] Master 127.15.93.190:43173 requested a full tablet report, sending...
I20260927 16:28:43.557590 23237 ts_manager.cc:194] Registered new tserver with Master: 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:43.558398 23237 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:50527
I20260927 16:28:43.562428 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23449
I20260927 16:28:43.562574 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:43.562924 15734 external_mini_cluster.cc:1428] Setting key 211271ad6579a1151cf4a364ee9f56d1
I20260927 16:28:43.564805 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.557894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.130:60403)
0927 16:28:43.558178 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.558192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.558278 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.558494 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.558497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.558708 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.558823 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.558831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.559825 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.559827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.560384 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.560392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.560781 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.560783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.560848 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.561328 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.561339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.561634 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.563359 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.563362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.563365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.563644 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.563897 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.563899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.563900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.563971 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.564402 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.564404 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.564477 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.564559 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.564670 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260927 16:28:43.564949 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.557975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:60403 (local address 127.15.93.190:43173)
0927 16:28:43.558076 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.558080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.558254 (+ 174us) server_negotiation.cc:407] Connection header received
0927 16:28:43.558351 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.558353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.558385 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.558446 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.558941 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.559699 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.560484 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.560689 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.560765 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.561736 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.561738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.561740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.561755 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.563245 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.563726 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.563729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.563730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.563811 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.564058 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.564061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.564063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.564305 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.564403 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.564637 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.564750 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for oryx/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
I20260927 16:28:43.590240 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:43173
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:43.697332 23589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:43.697593 23589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:43.697678 23589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:43.701591 23589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:43.701676 23589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:43.701748 23589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 16:28:43.701826 23589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:43.707072 23589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.93.190:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:43.708103 23589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:43.709076 23589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:43.718291 23594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:43.718326 23595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.718457 23589 server_base.cc:1034] running on GCE node
W20260927 16:28:43.718326 23597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:43.718921 23589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:43.719615 23589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:43.720798 23589 hybrid_clock.cc:648] HybridClock initialized: now 1790526523720784 us; error 25 us; skew 500 ppm
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:43.723577 23589 init.cc:377] Logged in from keytab as oryx/127.15.93.131@KRBTEST.COM (short username oryx)
I20260927 16:28:43.724426 23589 webserver.cc:460] Webserver started at http://127.15.93.131:40541/ using document root <none> and password file <none>
I20260927 16:28:43.724970 23589 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:43.725076 23589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:43.725366 23589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:43.727494 23589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "95cb20409fc942b694c73ea5b96de915"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "56bab89330f72e8f11ee00324edb03e7"
server_key_iv: "cb2c9f22b6412ed114376f19e6c0a173"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.728049 23589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "95cb20409fc942b694c73ea5b96de915"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "56bab89330f72e8f11ee00324edb03e7"
server_key_iv: "cb2c9f22b6412ed114376f19e6c0a173"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.731559 23589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:28:43.733843 23604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.735056 23589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:43.735193 23589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "95cb20409fc942b694c73ea5b96de915"
format_stamp: "Formatted at 2026-09-27 16:28:43 on dist-test-slave-grzf"
server_key: "56bab89330f72e8f11ee00324edb03e7"
server_key_iv: "cb2c9f22b6412ed114376f19e6c0a173"
server_key_version: "encryptionkey@0"
I20260927 16:28:43.735298 23589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:43.758602 23589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:43.761900 23589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:43.762117 23589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:43.763566 23589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:43.763634 23589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.763698 23589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:43.763751 23589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:43.774120 23589 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:41835
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:43.774420 23719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:41835 every 8 connection(s)
I20260927 16:28:43.775158 23589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:43.776173 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23589
I20260927 16:28:43.776305 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:43.776575 15734 external_mini_cluster.cc:1428] Setting key 7c9092b91add04a53bc42a1864f129cd
I20260927 16:28:43.779521 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.764340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:37765 (local address 127.15.93.190:43173)
0927 16:28:43.764486 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.764491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.765400 (+ 909us) server_negotiation.cc:407] Connection header received
0927 16:28:43.766570 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.766574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.766649 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.766767 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.768425 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.769290 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.771428 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.771732 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.771817 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.774842 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.774846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.774849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.774877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.777538 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.778036 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.778039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.778041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.778127 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.778460 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.778465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.778466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.778652 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.778758 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.779221 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.779333 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 16:28:43.780257 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.764807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.131:37765)
0927 16:28:43.765231 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.765280 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.766305 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.767001 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.767015 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.767505 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.768234 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.768259 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.769426 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.769431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.771252 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.771286 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.771887 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.771893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.772120 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.772807 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.772833 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.774680 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.777651 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.777657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.777670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.777922 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.778219 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.778223 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.778225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.778364 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.778756 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.778763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.778990 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.779170 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.779488 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":123,"threads_started":1}
I20260927 16:28:43.781356 23720 heartbeater.cc:347] Connected to a master server at 127.15.93.190:43173
I20260927 16:28:43.781576 23720 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:43.782064 23720 heartbeater.cc:511] Master 127.15.93.190:43173 requested a full tablet report, sending...
I20260927 16:28:43.783046 23236 ts_manager.cc:194] Registered new tserver with Master: 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835)
I20260927 16:28:43.783751 23236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:37765
I20260927 16:28:43.790457 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:28:43.792332 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.783785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:59273 (local address 127.15.93.190:43173)
0927 16:28:43.783899 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.783903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.784180 (+ 277us) server_negotiation.cc:407] Connection header received
0927 16:28:43.784270 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.784273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.784330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.784411 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.784895 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.785408 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.786264 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.786458 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.786532 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.787679 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.787682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.787685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.787705 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.789394 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.789914 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.789916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.789917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.789991 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.790307 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.790309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.790311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.791640 (+ 1329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.791770 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.792061 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.792165 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:43.792346 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.783668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.15.93.131:59273)
0927 16:28:43.784079 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.784094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.784196 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.784457 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.784460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.784672 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:43.784777 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.784784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.785539 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.785542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.786166 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.786175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.786552 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.786554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.786646 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.787168 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:43.787179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:43.787545 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:43.789527 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.789531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.789533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.789789 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.790095 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:43.790099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:43.790101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:43.790219 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:43.791771 (+ 1552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:43.791775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:43.791867 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.792001 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.792154 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":103,"threads_started":1}
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:28:43.813968 23286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:43.808136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.15.93.190:43173)
0927 16:28:43.808252 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.808255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.808267 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:43.808301 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.808303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.808339 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.808429 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.809161 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.809656 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.810531 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.810752 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.810860 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.813858 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58592: BlockingRecv error: recv got EOF from 127.0.0.1:58592 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 16:28:43 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:43.823024 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.815740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.15.93.190:43173)
0927 16:28:43.815854 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.815857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.815952 (+ 95us) server_negotiation.cc:407] Connection header received
0927 16:28:43.816045 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.816047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.816080 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.816156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:43.816699 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.817183 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.817892 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.818084 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.818163 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.820383 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:43.820386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:43.820388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:43.820411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:43.821655 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.822026 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.822029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.822030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.822105 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:43.822325 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:43.822327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:43.822328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:43.822492 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:43.822594 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.822811 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.822907 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:28:43.825786 23236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:43.829358 23236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:43.844430 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.838149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32865 (local address 127.0.0.1:58002)
0927 16:28:43.840149 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.840163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.840251 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.840666 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.840669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.840680 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:43.840793 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.840799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.842013 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.842015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.842821 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.842829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.843892 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.843896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.844000 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.844039 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.844151 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.844273 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260927 16:28:43.844430 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.839011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.15.93.129:32865)
0927 16:28:43.839267 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.839272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.840229 (+ 957us) server_negotiation.cc:407] Connection header received
0927 16:28:43.840374 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.840379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.840543 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.840698 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:43.840884 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.841906 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.842945 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.843769 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.843883 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.844036 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.844197 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.844297 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260927 16:28:43.845042 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.837857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.0.0.1:35720)
0927 16:28:43.839360 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.839412 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.839511 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.840020 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.840028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.840055 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:43.840197 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.840208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.841600 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.841603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.843020 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.843032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.844246 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.844250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.844334 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.844755 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.844823 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.844896 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"mutex_wait_us":39,"thread_start_us":81,"threads_started":1}
I20260927 16:28:43.845458 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.838654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.15.93.131:41835)
0927 16:28:43.838977 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.838992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.839489 (+ 497us) server_negotiation.cc:407] Connection header received
0927 16:28:43.839667 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.839674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.839855 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.839999 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:43.840310 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.841469 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.843153 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.843927 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.844020 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.844186 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.845253 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.845331 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
I20260927 16:28:43.845856 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.837840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:40249 (local address 127.0.0.1:38572)
0927 16:28:43.838769 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:43.838821 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:43.838944 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:43.840644 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:43.840649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:43.840667 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:43.840794 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.840799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.843270 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.843275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.844217 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:43.844226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.845486 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.845489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:43.845576 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.845597 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:28:43.845669 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 16:28:43.845739 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":27,"thread_start_us":62,"threads_started":1}
I20260927 16:28:43.845961 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:43.838005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.15.93.130:40249)
0927 16:28:43.839608 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:43.839619 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:43.839641 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:28:43.839713 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:43.839718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:43.839913 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:43.840420 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:43.842053 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.843157 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.844343 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:43.845180 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:43.845276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:43.845413 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:43.845723 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 16:28:43.845827 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495,"thread_start_us":130,"threads_started":1}
I20260927 16:28:43.847961 23654 tablet_service.cc:1467] Processing CreateTablet for tablet 4166bd64f89646e894257e2337c9deb2 (DEFAULT_TABLE table=test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:43.847952 23374 tablet_service.cc:1467] Processing CreateTablet for tablet 4166bd64f89646e894257e2337c9deb2 (DEFAULT_TABLE table=test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:43.848203 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 4166bd64f89646e894257e2337c9deb2 (DEFAULT_TABLE table=test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:43.849032 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166bd64f89646e894257e2337c9deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:43.849036 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166bd64f89646e894257e2337c9deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:43.849036 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4166bd64f89646e894257e2337c9deb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:43.855847 23748 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Bootstrap starting.
I20260927 16:28:43.857326 23750 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Bootstrap starting.
I20260927 16:28:43.857326 23749 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Bootstrap starting.
I20260927 16:28:43.858105 23748 tablet_bootstrap.cc:654] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:43.858814 23748 log.cc:824] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:43.859413 23750 tablet_bootstrap.cc:654] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:43.859412 23749 tablet_bootstrap.cc:654] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:43.860241 23749 log.cc:824] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:43.860241 23750 log.cc:824] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:43.860677 23748 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: No bootstrap required, opened a new log
I20260927 16:28:43.860879 23748 ts_tablet_manager.cc:1397] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 16:28:43.862141 23749 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: No bootstrap required, opened a new log
I20260927 16:28:43.862334 23749 ts_tablet_manager.cc:1397] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:28:43.862655 23750 tablet_bootstrap.cc:492] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: No bootstrap required, opened a new log
I20260927 16:28:43.862824 23750 ts_tablet_manager.cc:1397] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:28:43.864845 23748 raft_consensus.cc:348] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.865121 23748 raft_consensus.cc:374] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:43.865177 23748 raft_consensus.cc:729] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d414f14454310828190f716607b99, State: Initialized, Role: FOLLOWER
I20260927 16:28:43.865619 23748 consensus_queue.cc:260] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.865862 23749 raft_consensus.cc:348] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.866135 23749 raft_consensus.cc:374] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:43.866196 23749 raft_consensus.cc:729] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Initialized, Role: FOLLOWER
I20260927 16:28:43.866456 23580 heartbeater.cc:503] Master 127.15.93.190:43173 was elected leader, sending a full tablet report...
I20260927 16:28:43.866640 23749 consensus_queue.cc:260] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.866581 23750 raft_consensus.cc:348] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.866811 23750 raft_consensus.cc:374] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:43.866861 23750 raft_consensus.cc:729] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Initialized, Role: FOLLOWER
I20260927 16:28:43.867254 23440 heartbeater.cc:503] Master 127.15.93.190:43173 was elected leader, sending a full tablet report...
I20260927 16:28:43.867255 23750 consensus_queue.cc:260] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:43.867746 23749 ts_tablet_manager.cc:1428] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260927 16:28:43.868140 23750 ts_tablet_manager.cc:1428] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 16:28:43.868520 23720 heartbeater.cc:503] Master 127.15.93.190:43173 was elected leader, sending a full tablet report...
I20260927 16:28:43.869608 23748 ts_tablet_manager.cc:1428] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260927 16:28:44.020731 23754 raft_consensus.cc:482] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:44.020978 23754 raft_consensus.cc:504] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:44.022541 23754 leader_election.cc:288] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835), 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865)
W20260927 16:28:44.026484 23721 tablet.cc:2369] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:44.026978 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.022782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.15.93.130:51951)
0927 16:28:44.022987 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.023003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.023101 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.023465 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.023469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.023492 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.023624 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.023630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.024494 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.024500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.025512 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.025522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.026556 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.026559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.026681 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.026704 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.026767 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.026840 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 16:28:44.027433 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.022973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:51951 (local address 127.15.93.131:41835)
0927 16:28:44.023121 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.023127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.023187 (+ 60us) server_negotiation.cc:407] Connection header received
0927 16:28:44.023254 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.023259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.023323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.023451 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.023762 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.024383 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.025659 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.026339 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.026450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.026501 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.027221 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.027297 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:28:44.027969 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166bd64f89646e894257e2337c9deb2" candidate_uuid: "4d4d414f14454310828190f716607b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cb20409fc942b694c73ea5b96de915" is_pre_election: true
I20260927 16:28:44.028263 23674 raft_consensus.cc:2454] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d4d414f14454310828190f716607b99 in term 0.
I20260927 16:28:44.028477 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.022979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:43641 (local address 127.15.93.129:32865)
0927 16:28:44.023109 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.023126 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.023550 (+ 424us) server_negotiation.cc:407] Connection header received
0927 16:28:44.023764 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.023769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.023859 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.023940 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.025019 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.025764 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.027087 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.027733 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.027838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.027882 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.028201 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.028308 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:28:44.028808 23467 leader_election.cc:302] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4d414f14454310828190f716607b99, 95cb20409fc942b694c73ea5b96de915; no voters:
I20260927 16:28:44.028982 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.022782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32865 (local address 127.15.93.130:43641)
0927 16:28:44.023460 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.023472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.023613 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.024073 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.024078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.024102 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.024868 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.024881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.025886 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.025889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.026953 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.026961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.027871 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.027876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.028020 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.028058 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.028637 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.028736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":357,"thread_start_us":107,"threads_started":1}
I20260927 16:28:44.029157 23754 raft_consensus.cc:2781] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:44.029254 23754 raft_consensus.cc:482] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:44.029318 23754 raft_consensus.cc:3037] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.029556 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166bd64f89646e894257e2337c9deb2" candidate_uuid: "4d4d414f14454310828190f716607b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" is_pre_election: true
I20260927 16:28:44.029814 23394 raft_consensus.cc:2454] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d4d414f14454310828190f716607b99 in term 0.
I20260927 16:28:44.030989 23754 raft_consensus.cc:504] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:44.031466 23754 leader_election.cc:288] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [CANDIDATE]: Term 1 election: Requested vote from peers 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835), 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865)
I20260927 16:28:44.031747 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166bd64f89646e894257e2337c9deb2" candidate_uuid: "4d4d414f14454310828190f716607b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cb20409fc942b694c73ea5b96de915"
I20260927 16:28:44.031800 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166bd64f89646e894257e2337c9deb2" candidate_uuid: "4d4d414f14454310828190f716607b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
I20260927 16:28:44.031874 23674 raft_consensus.cc:3037] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.031934 23394 raft_consensus.cc:3037] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.033181 23674 raft_consensus.cc:2454] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4d414f14454310828190f716607b99 in term 1.
I20260927 16:28:44.033458 23467 leader_election.cc:302] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4d414f14454310828190f716607b99, 95cb20409fc942b694c73ea5b96de915; no voters:
I20260927 16:28:44.033537 23394 raft_consensus.cc:2454] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4d414f14454310828190f716607b99 in term 1.
I20260927 16:28:44.033650 23754 raft_consensus.cc:2781] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:44.033856 23754 raft_consensus.cc:686] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 1 LEADER]: Becoming Leader. State: Replica: 4d4d414f14454310828190f716607b99, State: Running, Role: LEADER
I20260927 16:28:44.034206 23754 consensus_queue.cc:237] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } }
I20260927 16:28:44.037546 23236 catalog_manager.cc:5495] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d4d414f14454310828190f716607b99 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "4d4d414f14454310828190f716607b99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } health_report { overall_health: UNKNOWN } } }
W20260927 16:28:44.053808 23581 tablet.cc:2369] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:28:44.071798 23441 tablet.cc:2369] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:28:44.079635 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.15.93.130:40249)
0927 16:28:44.076107 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.076111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.076135 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:28:44.076272 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.076275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.076342 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.076460 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.076702 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.077298 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.078216 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.078459 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.078565 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.078832 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.079241 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.079393 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.079442 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.079496 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:28:44.085076 23394 raft_consensus.cc:1261] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Refusing update from remote peer 4d4d414f14454310828190f716607b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.085132 23674 raft_consensus.cc:1261] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Refusing update from remote peer 4d4d414f14454310828190f716607b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.085812 23754 consensus_queue.cc:1035] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.086268 23758 consensus_queue.cc:1035] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.095548 23762 mvcc.cc:205] Tried to move back new op lower bound from 7333996642645872640 to 7333996642446761984. Current Snapshot: MvccSnapshot[applied={T|T < 7333996642645872640}]
I20260927 16:28:44.096761 23763 mvcc.cc:205] Tried to move back new op lower bound from 7333996642645872640 to 7333996642446761984. Current Snapshot: MvccSnapshot[applied={T|T < 7333996642645872640}]
I20260927 16:28:44.104010 23237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.104180 23237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.106580 23237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Sending DeleteTablet for 3 replicas of tablet 4166bd64f89646e894257e2337c9deb2
I20260927 16:28:44.107249 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166bd64f89646e894257e2337c9deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260927 16:28:44.107738 23654 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166bd64f89646e894257e2337c9deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35720
I20260927 16:28:44.107838 23771 tablet_replica.cc:330] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: stopping tablet replica
I20260927 16:28:44.108206 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166bd64f89646e894257e2337c9deb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 16:28:44.108255 23771 raft_consensus.cc:2229] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:44.108724 23771 raft_consensus.cc:2258] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.109393 23237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:44.109804 23237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:44.110431 23771 ts_tablet_manager.cc:1905] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.113394 23771 ts_tablet_manager.cc:1918] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.113555 23771 log.cc:1196] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/4166bd64f89646e894257e2337c9deb2
I20260927 16:28:44.113965 23771 ts_tablet_manager.cc:1939] T 4166bd64f89646e894257e2337c9deb2 P 4d4d414f14454310828190f716607b99: Deleting consensus metadata
I20260927 16:28:44.114877 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 26a780b03b3641ae939a149d65b8eb30 (DEFAULT_TABLE table=test-table [id=660e8d71aa1b45a982b752fbc50442ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.115116 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a780b03b3641ae939a149d65b8eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.115283 23374 tablet_service.cc:1467] Processing CreateTablet for tablet 26a780b03b3641ae939a149d65b8eb30 (DEFAULT_TABLE table=test-table [id=660e8d71aa1b45a982b752fbc50442ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.115556 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a780b03b3641ae939a149d65b8eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.117290 23748 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Bootstrap starting.
I20260927 16:28:44.117916 23749 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Bootstrap starting.
I20260927 16:28:44.118264 23748 tablet_bootstrap.cc:654] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.118378 23654 tablet_service.cc:1467] Processing CreateTablet for tablet 26a780b03b3641ae939a149d65b8eb30 (DEFAULT_TABLE table=test-table [id=660e8d71aa1b45a982b752fbc50442ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.118633 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a780b03b3641ae939a149d65b8eb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.118829 23749 tablet_bootstrap.cc:654] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.119975 23224 catalog_manager.cc:4841] TS 4d4d414f14454310828190f716607b99 (127.15.93.130:40249): tablet 4166bd64f89646e894257e2337c9deb2 (table test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]) successfully deleted
I20260927 16:28:44.120846 23748 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: No bootstrap required, opened a new log
I20260927 16:28:44.121115 23750 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Bootstrap starting.
I20260927 16:28:44.122054 23750 tablet_bootstrap.cc:654] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.122603 23772 tablet_replica.cc:330] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: stopping tablet replica
I20260927 16:28:44.122700 23748 ts_tablet_manager.cc:1397] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 16:28:44.122923 23772 raft_consensus.cc:2229] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.123144 23748 raft_consensus.cc:348] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.123242 23748 raft_consensus.cc:374] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.123245 23772 raft_consensus.cc:2258] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.123276 23748 raft_consensus.cc:729] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d414f14454310828190f716607b99, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.123422 23748 consensus_queue.cc:260] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.123607 23773 tablet_replica.cc:330] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: stopping tablet replica
I20260927 16:28:44.123695 23748 ts_tablet_manager.cc:1428] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.125406 23750 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: No bootstrap required, opened a new log
I20260927 16:28:44.125491 23750 ts_tablet_manager.cc:1397] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 16:28:44.124636 23773 raft_consensus.cc:2229] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.125895 23773 raft_consensus.cc:2258] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.125914 23750 raft_consensus.cc:348] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.126011 23750 raft_consensus.cc:374] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.126040 23750 raft_consensus.cc:729] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.126152 23750 consensus_queue.cc:260] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.126401 23750 ts_tablet_manager.cc:1428] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:44.126967 23749 tablet_bootstrap.cc:492] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: No bootstrap required, opened a new log
I20260927 16:28:44.127063 23749 ts_tablet_manager.cc:1397] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 16:28:44.127512 23749 raft_consensus.cc:348] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.127615 23749 raft_consensus.cc:374] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.127646 23749 raft_consensus.cc:729] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.127750 23749 consensus_queue.cc:260] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.127974 23749 ts_tablet_manager.cc:1428] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.129201 23772 ts_tablet_manager.cc:1905] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.131361 23772 ts_tablet_manager.cc:1918] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.131500 23772 log.cc:1196] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/4166bd64f89646e894257e2337c9deb2
I20260927 16:28:44.131817 23772 ts_tablet_manager.cc:1939] T 4166bd64f89646e894257e2337c9deb2 P 95cb20409fc942b694c73ea5b96de915: Deleting consensus metadata
I20260927 16:28:44.132441 23773 ts_tablet_manager.cc:1905] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.132941 23223 catalog_manager.cc:4841] TS 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835): tablet 4166bd64f89646e894257e2337c9deb2 (table test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]) successfully deleted
I20260927 16:28:44.135584 23773 ts_tablet_manager.cc:1918] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.135689 23773 log.cc:1196] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/4166bd64f89646e894257e2337c9deb2
I20260927 16:28:44.135969 23773 ts_tablet_manager.cc:1939] T 4166bd64f89646e894257e2337c9deb2 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting consensus metadata
I20260927 16:28:44.136930 23221 catalog_manager.cc:4841] TS 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865): tablet 4166bd64f89646e894257e2337c9deb2 (table test-table [id=1c3681e71a5b4c37ad0b64307ecaad56]) successfully deleted
I20260927 16:28:44.329955 23755 raft_consensus.cc:482] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:44.330096 23755 raft_consensus.cc:504] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.331184 23755 leader_election.cc:288] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d4d414f14454310828190f716607b99 (127.15.93.130:40249), 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835)
I20260927 16:28:44.336193 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.331569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:53189 (local address 127.15.93.131:41835)
0927 16:28:44.331749 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.331754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.331849 (+ 95us) server_negotiation.cc:407] Connection header received
0927 16:28:44.332000 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.332004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.332052 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.332099 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.332440 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.333176 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.334894 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.335515 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.335641 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.335692 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.336002 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.336070 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 16:28:44.336196 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.331604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:50123 (local address 127.15.93.130:40249)
0927 16:28:44.331749 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.331754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.332080 (+ 326us) server_negotiation.cc:407] Connection header received
0927 16:28:44.332238 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.332242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.332294 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.332377 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.332629 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.333114 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.334894 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.335518 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.335641 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.335692 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.336001 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.336087 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 16:28:44.336262 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.331439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:40249 (local address 127.15.93.129:50123)
0927 16:28:44.331993 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.332004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.332130 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.332387 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.332389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.332405 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.332519 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.332524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.333219 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.333225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.334763 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.334773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.335655 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.335661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.335840 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.335890 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.335970 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.336067 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":511,"thread_start_us":75,"threads_started":1}
I20260927 16:28:44.336272 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.331439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.15.93.129:53189)
0927 16:28:44.331746 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.331760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.331869 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.332225 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.332228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.332242 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.332334 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.332339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.333315 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.333317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.334765 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.334773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.335663 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.335666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.335835 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.335894 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.335970 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.336067 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":168,"thread_start_us":76,"threads_started":1}
I20260927 16:28:44.336727 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a780b03b3641ae939a149d65b8eb30" candidate_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cb20409fc942b694c73ea5b96de915" is_pre_election: true
I20260927 16:28:44.336776 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a780b03b3641ae939a149d65b8eb30" candidate_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99" is_pre_election: true
I20260927 16:28:44.336884 23674 raft_consensus.cc:2454] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dac72e7e33f45ea992cdf5cb59c90f5 in term 0.
I20260927 16:28:44.337009 23533 raft_consensus.cc:2454] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dac72e7e33f45ea992cdf5cb59c90f5 in term 0.
I20260927 16:28:44.337253 23327 leader_election.cc:302] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:44.337535 23755 raft_consensus.cc:2781] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:44.337612 23755 raft_consensus.cc:482] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:44.337641 23755 raft_consensus.cc:3037] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.338716 23755 raft_consensus.cc:504] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.339093 23755 leader_election.cc:288] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [CANDIDATE]: Term 1 election: Requested vote from peers 4d4d414f14454310828190f716607b99 (127.15.93.130:40249), 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835)
I20260927 16:28:44.339422 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a780b03b3641ae939a149d65b8eb30" candidate_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99"
I20260927 16:28:44.339506 23674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a780b03b3641ae939a149d65b8eb30" candidate_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95cb20409fc942b694c73ea5b96de915"
I20260927 16:28:44.339573 23533 raft_consensus.cc:3037] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.339600 23674 raft_consensus.cc:3037] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.340668 23674 raft_consensus.cc:2454] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dac72e7e33f45ea992cdf5cb59c90f5 in term 1.
I20260927 16:28:44.340668 23533 raft_consensus.cc:2454] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dac72e7e33f45ea992cdf5cb59c90f5 in term 1.
I20260927 16:28:44.340992 23328 leader_election.cc:302] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4d414f14454310828190f716607b99, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:44.341167 23755 raft_consensus.cc:2781] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:44.341408 23755 raft_consensus.cc:686] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 LEADER]: Becoming Leader. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Running, Role: LEADER
I20260927 16:28:44.341655 23755 consensus_queue.cc:237] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.343765 23237 catalog_manager.cc:5495] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } health_report { overall_health: UNKNOWN } } }
Sep 27 16:28:44 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526523, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.190@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:44.389376 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.380951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:43173 (local address 127.0.0.1:58614)
0927 16:28:44.381345 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.381358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.381495 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.381776 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.381779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.382232 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:44.382318 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.382324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.383071 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.383077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.383699 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.383707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.384140 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.384144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.384278 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.384765 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:44.384777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:44.386141 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:44.387746 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:44.387757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:44.387764 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:44.387982 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:44.388251 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:44.388256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:44.388258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:44.388347 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:44.388767 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:44.388776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:44.388881 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.388993 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.389153 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":35,"thread_start_us":115,"threads_started":1}
I20260927 16:28:44.389394 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.380989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.15.93.190:43173)
0927 16:28:44.381280 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.381289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.381432 (+ 143us) server_negotiation.cc:407] Connection header received
0927 16:28:44.381596 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.381599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.381667 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.381760 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:44.382433 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.382967 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.383809 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.384029 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.384134 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.386272 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:44.386278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:44.386285 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:44.386304 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:44.387640 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:44.388078 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:44.388084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:44.388086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:44.388163 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:44.388447 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:44.388450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:44.388451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:44.388663 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:44.388757 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.389046 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.389208 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260927 16:28:44.393095 23237 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 16:28:44.393839 23237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:44.398557 23374 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfa13056bb46138a7c1e3bb85121a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a76723039e064653a5cb4db338254e9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:44.398590 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfa13056bb46138a7c1e3bb85121a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a76723039e064653a5cb4db338254e9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:44.398716 23654 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddfa13056bb46138a7c1e3bb85121a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a76723039e064653a5cb4db338254e9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 16:28:44.398968 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfa13056bb46138a7c1e3bb85121a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.398974 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfa13056bb46138a7c1e3bb85121a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.399024 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ddfa13056bb46138a7c1e3bb85121a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.401477 23749 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: Bootstrap starting.
I20260927 16:28:44.401609 23748 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: Bootstrap starting.
I20260927 16:28:44.401633 23750 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: Bootstrap starting.
I20260927 16:28:44.402442 23750 tablet_bootstrap.cc:654] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.402451 23749 tablet_bootstrap.cc:654] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.402452 23748 tablet_bootstrap.cc:654] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.403581 23750 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: No bootstrap required, opened a new log
I20260927 16:28:44.403664 23750 ts_tablet_manager.cc:1397] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 16:28:44.403723 23749 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: No bootstrap required, opened a new log
I20260927 16:28:44.403797 23749 ts_tablet_manager.cc:1397] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:44.403927 23748 tablet_bootstrap.cc:492] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: No bootstrap required, opened a new log
I20260927 16:28:44.403985 23748 ts_tablet_manager.cc:1397] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:44.404089 23750 raft_consensus.cc:348] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404214 23750 raft_consensus.cc:374] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.404181 23749 raft_consensus.cc:348] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404245 23750 raft_consensus.cc:729] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.404276 23749 raft_consensus.cc:374] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.404310 23749 raft_consensus.cc:729] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.404271 23748 raft_consensus.cc:348] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404384 23748 raft_consensus.cc:374] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.404377 23750 consensus_queue.cc:260] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404426 23748 raft_consensus.cc:729] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d414f14454310828190f716607b99, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.404420 23749 consensus_queue.cc:260] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404551 23748 consensus_queue.cc:260] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.404671 23749 ts_tablet_manager.cc:1428] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.404640 23750 ts_tablet_manager.cc:1428] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:44.404624 23756 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.404767 23756 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.404820 23748 ts_tablet_manager.cc:1428] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.405205 23766 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.405332 23766 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.406302 23755 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.406425 23755 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.461956 23756 raft_consensus.cc:482] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:44.462122 23756 raft_consensus.cc:504] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.463251 23756 leader_election.cc:288] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:44.468286 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.463564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:40249 (local address 127.15.93.131:49093)
0927 16:28:44.463949 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.463959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.464041 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.464259 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.464262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.464273 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.464363 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.464372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.465360 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.465364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.466929 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.466940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.467846 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.467849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.467957 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.467979 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.468042 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.468098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":150,"thread_start_us":65,"threads_started":1}
I20260927 16:28:44.468295 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.463565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:49093 (local address 127.15.93.130:40249)
0927 16:28:44.463821 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.463824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.464033 (+ 209us) server_negotiation.cc:407] Connection header received
0927 16:28:44.464118 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.464123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.464172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.464241 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.464468 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.465245 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.467052 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.467722 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.467841 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.467893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.468081 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.468170 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260927 16:28:44.468595 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.463578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:58425 (local address 127.15.93.129:32865)
0927 16:28:44.463727 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.463730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.463850 (+ 120us) server_negotiation.cc:407] Connection header received
0927 16:28:44.464003 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.464008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.464061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.464145 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:28:44.464468 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.465246 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.467052 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.468063 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.468189 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.468234 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.468444 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.468508 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 16:28:44.468942 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.463455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32865 (local address 127.15.93.131:58425)
0927 16:28:44.463744 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.463760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.463892 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.464175 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.464177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.464190 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:28:44.464335 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.464344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.465360 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.465364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.466930 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.466940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.468181 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.468184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.468276 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.468304 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.468753 (+ 449us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.468843 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":154,"thread_start_us":96,"threads_started":1}
I20260927 16:28:44.468899 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddfa13056bb46138a7c1e3bb85121a8" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99" is_pre_election: true
I20260927 16:28:44.469060 23533 raft_consensus.cc:2454] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:44.469390 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddfa13056bb46138a7c1e3bb85121a8" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" is_pre_election: true
I20260927 16:28:44.469405 23608 leader_election.cc:302] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4d414f14454310828190f716607b99, 95cb20409fc942b694c73ea5b96de915; no voters:
I20260927 16:28:44.469532 23394 raft_consensus.cc:2454] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:44.469666 23756 raft_consensus.cc:2781] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:44.469724 23756 raft_consensus.cc:482] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:44.469774 23756 raft_consensus.cc:3037] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.470722 23756 raft_consensus.cc:504] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.471058 23756 leader_election.cc:288] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Requested vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:44.471443 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddfa13056bb46138a7c1e3bb85121a8" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
I20260927 16:28:44.471428 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ddfa13056bb46138a7c1e3bb85121a8" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99"
I20260927 16:28:44.471547 23394 raft_consensus.cc:3037] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.471547 23533 raft_consensus.cc:3037] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.472596 23533 raft_consensus.cc:2454] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:44.472596 23394 raft_consensus.cc:2454] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:44.472862 23605 leader_election.cc:302] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:44.473021 23756 raft_consensus.cc:2781] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:44.473220 23756 raft_consensus.cc:686] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [term 1 LEADER]: Becoming Leader. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Running, Role: LEADER
I20260927 16:28:44.473464 23756 consensus_queue.cc:237] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.475217 23793 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: TxnStatusTablet state changed. Reason: New leader 95cb20409fc942b694c73ea5b96de915. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.475323 23793 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:44.475817 23795 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 16:28:44.475966 23795 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 16:28:44.476305 23235 catalog_manager.cc:5495] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95cb20409fc942b694c73ea5b96de915 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } health_report { overall_health: HEALTHY } } }
I20260927 16:28:44.506151 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.502343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.0.0.1:35730)
0927 16:28:44.502515 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.502531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.502664 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.502990 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.502993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.503000 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:44.503107 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.503112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.503929 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.503932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.504523 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.504530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.505036 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.505040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.505142 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.505202 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.505834 (+ 632us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.505849 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.505942 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.506026 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 16:28:44.506172 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.502452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.15.93.131:41835)
0927 16:28:44.502578 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.502583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.502597 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:44.502821 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.502823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.502877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.502978 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.503228 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.503810 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.504641 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.504926 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.505050 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.505342 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.505739 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.505858 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.505965 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.506017 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:28:44.509284 23394 raft_consensus.cc:1261] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.509320 23533 raft_consensus.cc:1261] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.509810 23793 consensus_queue.cc:1035] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.510044 23756 consensus_queue.cc:1035] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.511718 23755 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: TxnStatusTablet state changed. Reason: New leader 95cb20409fc942b694c73ea5b96de915. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.511837 23755 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.513298 23766 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: TxnStatusTablet state changed. Reason: New leader 95cb20409fc942b694c73ea5b96de915. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.513406 23766 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.513640 23793 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.513741 23793 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:44.514014 23793 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.514103 23793 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 95cb20409fc942b694c73ea5b96de915: This TxnStatusTablet replica's current role is: LEADER
I20260927 16:28:44.517055 23755 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.517166 23755 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 9dac72e7e33f45ea992cdf5cb59c90f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.518117 23766 tablet_replica.cc:439] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } } }
I20260927 16:28:44.518240 23766 tablet_replica.cc:442] T 9ddfa13056bb46138a7c1e3bb85121a8 P 4d4d414f14454310828190f716607b99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 16:28:44.523489 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.519866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.15.93.129:32865)
0927 16:28:44.519971 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.519975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.519993 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:28:44.520123 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.520127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.520184 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.520286 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.520468 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.521232 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.522011 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.522350 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.522448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.522659 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.523147 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.523257 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.523293 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.523346 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:28:44.533075 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.529788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:59491 (local address 127.15.93.131:41835)
0927 16:28:44.529907 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.529911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.530031 (+ 120us) server_negotiation.cc:407] Connection header received
0927 16:28:44.530161 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.530163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.530204 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.530256 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.530494 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.530943 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.532154 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.532354 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.532430 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.532720 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.532778 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.532829 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.532946 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.532985 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:28:44.533144 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.529662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.15.93.129:59491)
0927 16:28:44.529967 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.529981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.530076 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.530287 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.530292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.530302 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:44.530408 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.530414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.531035 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.531038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.532040 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.532051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.532455 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.532459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.532551 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.532596 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.532854 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.532860 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.532928 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.532996 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
I20260927 16:28:44.539820 23533 raft_consensus.cc:1261] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Refusing update from remote peer 9dac72e7e33f45ea992cdf5cb59c90f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.539880 23674 raft_consensus.cc:1261] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Refusing update from remote peer 9dac72e7e33f45ea992cdf5cb59c90f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.540295 23755 consensus_queue.cc:1035] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.540654 23755 consensus_queue.cc:1035] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.544512 23761 mvcc.cc:205] Tried to move back new op lower bound from 7333996644511158272 to 7333996643705659392. Current Snapshot: MvccSnapshot[applied={T|T < 7333996644511158272}]
I20260927 16:28:44.565230 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.561094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:36101 (local address 127.15.93.129:32865)
0927 16:28:44.561316 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.561320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.561411 (+ 91us) server_negotiation.cc:407] Connection header received
0927 16:28:44.561549 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.561552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.561598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.561680 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.561925 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.562741 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.564013 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.564341 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.564409 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.564803 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.564885 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.564961 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.565059 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.565107 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260927 16:28:44.565271 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.561038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32865 (local address 127.15.93.131:36101)
0927 16:28:44.561340 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:44.561355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:44.561453 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:44.561698 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:44.561701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:44.561719 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:44.561828 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.561834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.562854 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.562857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.563876 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:44.563887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.564528 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.564531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:44.564617 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.564670 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.564978 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.564984 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:28:44.565050 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 16:28:44.565123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260927 16:28:44.589515 23235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.589681 23235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.591837 23235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Sending DeleteTablet for 3 replicas of tablet 26a780b03b3641ae939a149d65b8eb30
I20260927 16:28:44.592317 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a780b03b3641ae939a149d65b8eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 16:28:44.592509 23773 tablet_replica.cc:330] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: stopping tablet replica
I20260927 16:28:44.592638 23773 raft_consensus.cc:2229] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:44.592703 23654 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a780b03b3641ae939a149d65b8eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35720
I20260927 16:28:44.592718 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a780b03b3641ae939a149d65b8eb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260927 16:28:44.592870 23772 tablet_replica.cc:330] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: stopping tablet replica
I20260927 16:28:44.592880 23771 tablet_replica.cc:330] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: stopping tablet replica
I20260927 16:28:44.592911 23773 raft_consensus.cc:2258] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.592975 23772 raft_consensus.cc:2229] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.592993 23771 raft_consensus.cc:2229] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.593072 23772 raft_consensus.cc:2258] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.593093 23771 raft_consensus.cc:2258] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.593803 23772 ts_tablet_manager.cc:1905] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.593832 23771 ts_tablet_manager.cc:1905] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.595914 23773 ts_tablet_manager.cc:1905] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.596402 23772 ts_tablet_manager.cc:1918] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:44.596467 23772 log.cc:1196] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/26a780b03b3641ae939a149d65b8eb30
I20260927 16:28:44.596611 23771 ts_tablet_manager.cc:1918] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:44.596690 23771 log.cc:1196] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/26a780b03b3641ae939a149d65b8eb30
I20260927 16:28:44.596738 23772 ts_tablet_manager.cc:1939] T 26a780b03b3641ae939a149d65b8eb30 P 95cb20409fc942b694c73ea5b96de915: Deleting consensus metadata
I20260927 16:28:44.596948 23771 ts_tablet_manager.cc:1939] T 26a780b03b3641ae939a149d65b8eb30 P 4d4d414f14454310828190f716607b99: Deleting consensus metadata
I20260927 16:28:44.597375 23223 catalog_manager.cc:4841] TS 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835): tablet 26a780b03b3641ae939a149d65b8eb30 (table test-table [id=660e8d71aa1b45a982b752fbc50442ff]) successfully deleted
I20260927 16:28:44.597534 23224 catalog_manager.cc:4841] TS 4d4d414f14454310828190f716607b99 (127.15.93.130:40249): tablet 26a780b03b3641ae939a149d65b8eb30 (table test-table [id=660e8d71aa1b45a982b752fbc50442ff]) successfully deleted
I20260927 16:28:44.598398 23773 ts_tablet_manager.cc:1918] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:44.598448 23773 log.cc:1196] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/26a780b03b3641ae939a149d65b8eb30
I20260927 16:28:44.598626 23773 ts_tablet_manager.cc:1939] T 26a780b03b3641ae939a149d65b8eb30 P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting consensus metadata
I20260927 16:28:44.599200 23221 catalog_manager.cc:4841] TS 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865): tablet 26a780b03b3641ae939a149d65b8eb30 (table test-table [id=660e8d71aa1b45a982b752fbc50442ff]) successfully deleted
Sep 27 16:28:44 dist-test-slave-grzf krb5kdc[23192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 16:28:44 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:44 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:28:44.624609 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:44.618278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.15.93.190:43173)
0927 16:28:44.618413 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.618419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.618785 (+ 366us) server_negotiation.cc:407] Connection header received
0927 16:28:44.618949 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.618952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.619008 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.619092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:44.619865 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.620400 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.621255 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.621489 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.621593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.624511 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58622: BlockingRecv error: recv got EOF from 127.0.0.1:58622 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 16:28:44 dist-test-slave-grzf krb5kdc[23192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM
I20260927 16:28:44.634802 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.626656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.15.93.190:43173)
0927 16:28:44.626755 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.626759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.627125 (+ 366us) server_negotiation.cc:407] Connection header received
0927 16:28:44.627266 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.627269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.627307 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.627423 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:44.628007 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.628489 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.629262 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.629480 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.629585 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.631767 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:44.631771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:44.631774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:44.631803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:44.633219 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:44.633630 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:44.633633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:44.633640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:44.633710 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:44.633977 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:44.633981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:44.633983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:44.634158 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:44.634294 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.634524 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.634637 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:28:44.637292 23235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:44.637741 23235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:44.642596 23374 tablet_service.cc:1467] Processing CreateTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d (DEFAULT_TABLE table=test-table [id=998853b981c8412f8b414c96c78ea484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.642905 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b84bba2e1d4743b0152f0de7764d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.642915 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d (DEFAULT_TABLE table=test-table [id=998853b981c8412f8b414c96c78ea484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.643170 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b84bba2e1d4743b0152f0de7764d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.645422 23749 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Bootstrap starting.
I20260927 16:28:44.645422 23748 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Bootstrap starting.
I20260927 16:28:44.646315 23749 tablet_bootstrap.cc:654] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.646315 23748 tablet_bootstrap.cc:654] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.647166 23654 tablet_service.cc:1467] Processing CreateTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d (DEFAULT_TABLE table=test-table [id=998853b981c8412f8b414c96c78ea484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.647451 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b84bba2e1d4743b0152f0de7764d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.647665 23749 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: No bootstrap required, opened a new log
I20260927 16:28:44.647738 23749 ts_tablet_manager.cc:1397] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:44.648126 23749 raft_consensus.cc:348] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.648223 23749 raft_consensus.cc:374] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.648250 23749 raft_consensus.cc:729] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.648355 23749 consensus_queue.cc:260] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.648639 23749 ts_tablet_manager.cc:1428] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.649729 23750 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Bootstrap starting.
I20260927 16:28:44.650512 23750 tablet_bootstrap.cc:654] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.651747 23748 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: No bootstrap required, opened a new log
I20260927 16:28:44.651829 23748 ts_tablet_manager.cc:1397] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 16:28:44.652010 23750 tablet_bootstrap.cc:492] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: No bootstrap required, opened a new log
I20260927 16:28:44.652078 23750 ts_tablet_manager.cc:1397] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:44.652215 23748 raft_consensus.cc:348] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.652310 23748 raft_consensus.cc:374] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.652364 23748 raft_consensus.cc:729] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d414f14454310828190f716607b99, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.652475 23750 raft_consensus.cc:348] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.652477 23748 consensus_queue.cc:260] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.652560 23750 raft_consensus.cc:374] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.652593 23750 raft_consensus.cc:729] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.652702 23750 consensus_queue.cc:260] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.652755 23748 ts_tablet_manager.cc:1428] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.652956 23750 ts_tablet_manager.cc:1428] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.887187 23756 raft_consensus.cc:482] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:44.887313 23756 raft_consensus.cc:504] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.887810 23756 leader_election.cc:288] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:44.888250 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b84bba2e1d4743b0152f0de7764d0d" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99" is_pre_election: true
I20260927 16:28:44.888250 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b84bba2e1d4743b0152f0de7764d0d" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" is_pre_election: true
I20260927 16:28:44.888445 23394 raft_consensus.cc:2454] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:44.888420 23533 raft_consensus.cc:2454] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:44.888751 23605 leader_election.cc:302] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:44.888909 23756 raft_consensus.cc:2781] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:44.888965 23756 raft_consensus.cc:482] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:44.889009 23756 raft_consensus.cc:3037] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.890007 23756 raft_consensus.cc:504] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.890367 23756 leader_election.cc:288] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Requested vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:44.890663 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b84bba2e1d4743b0152f0de7764d0d" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
I20260927 16:28:44.890798 23394 raft_consensus.cc:3037] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.890803 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b84bba2e1d4743b0152f0de7764d0d" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99"
I20260927 16:28:44.890910 23533 raft_consensus.cc:3037] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:44.891867 23394 raft_consensus.cc:2454] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:44.891952 23533 raft_consensus.cc:2454] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:44.892140 23605 leader_election.cc:302] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:44.892297 23756 raft_consensus.cc:2781] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:44.892409 23756 raft_consensus.cc:686] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 1 LEADER]: Becoming Leader. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Running, Role: LEADER
I20260927 16:28:44.892558 23756 consensus_queue.cc:237] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.894052 23236 catalog_manager.cc:5495] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95cb20409fc942b694c73ea5b96de915 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } health_report { overall_health: HEALTHY } } }
I20260927 16:28:44.949409 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:44.946388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.15.93.131:41835)
0927 16:28:44.946512 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:44.946517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:44.946531 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:44.946606 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:44.946609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:44.946661 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:44.946741 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:44.946990 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.947516 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.948378 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:44.948596 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:44.948700 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:44.948970 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:44.949037 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:44.949115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:44.949209 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 16:28:44.949262 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:28:44.951602 23533 raft_consensus.cc:1261] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.951637 23394 raft_consensus.cc:1261] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:44.952148 23756 consensus_queue.cc:1035] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.952283 23793 consensus_queue.cc:1035] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:44.962167 23235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.962318 23235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:44.964537 23235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Sending DeleteTablet for 3 replicas of tablet 93b84bba2e1d4743b0152f0de7764d0d
I20260927 16:28:44.965265 23654 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35720
I20260927 16:28:44.965276 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 16:28:44.965457 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet 93b84bba2e1d4743b0152f0de7764d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:44 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260927 16:28:44.965636 23771 tablet_replica.cc:330] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: stopping tablet replica
I20260927 16:28:44.965641 23772 tablet_replica.cc:330] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: stopping tablet replica
I20260927 16:28:44.965754 23772 raft_consensus.cc:2229] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:44.965755 23771 raft_consensus.cc:2229] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.965876 23771 raft_consensus.cc:2258] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.966055 23772 raft_consensus.cc:2258] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.966228 23773 tablet_replica.cc:330] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: stopping tablet replica
I20260927 16:28:44.966341 23773 raft_consensus.cc:2229] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:44.966429 23773 raft_consensus.cc:2258] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:44.966555 23771 ts_tablet_manager.cc:1905] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.967087 23773 ts_tablet_manager.cc:1905] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.967341 23235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:28:44.967737 23235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:28:44.969399 23772 ts_tablet_manager.cc:1905] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:44.969566 23773 ts_tablet_manager.cc:1918] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.969599 23771 ts_tablet_manager.cc:1918] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.969635 23773 log.cc:1196] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/93b84bba2e1d4743b0152f0de7764d0d
I20260927 16:28:44.969667 23771 log.cc:1196] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/93b84bba2e1d4743b0152f0de7764d0d
I20260927 16:28:44.969897 23773 ts_tablet_manager.cc:1939] T 93b84bba2e1d4743b0152f0de7764d0d P 9dac72e7e33f45ea992cdf5cb59c90f5: Deleting consensus metadata
I20260927 16:28:44.969915 23771 ts_tablet_manager.cc:1939] T 93b84bba2e1d4743b0152f0de7764d0d P 4d4d414f14454310828190f716607b99: Deleting consensus metadata
I20260927 16:28:44.970557 23221 catalog_manager.cc:4841] TS 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865): tablet 93b84bba2e1d4743b0152f0de7764d0d (table test-table [id=998853b981c8412f8b414c96c78ea484]) successfully deleted
I20260927 16:28:44.970976 23224 catalog_manager.cc:4841] TS 4d4d414f14454310828190f716607b99 (127.15.93.130:40249): tablet 93b84bba2e1d4743b0152f0de7764d0d (table test-table [id=998853b981c8412f8b414c96c78ea484]) successfully deleted
I20260927 16:28:44.971796 23772 ts_tablet_manager.cc:1918] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:28:44.971870 23772 log.cc:1196] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/93b84bba2e1d4743b0152f0de7764d0d
I20260927 16:28:44.972167 23772 ts_tablet_manager.cc:1939] T 93b84bba2e1d4743b0152f0de7764d0d P 95cb20409fc942b694c73ea5b96de915: Deleting consensus metadata
I20260927 16:28:44.972767 23223 catalog_manager.cc:4841] TS 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835): tablet 93b84bba2e1d4743b0152f0de7764d0d (table test-table [id=998853b981c8412f8b414c96c78ea484]) successfully deleted
I20260927 16:28:44.974701 23374 tablet_service.cc:1467] Processing CreateTablet for tablet ba65d8d08aab4c9da9d10cb214286943 (DEFAULT_TABLE table=test-table [id=f76a42b8c7ec4618b095c00c4fb8a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.974773 23654 tablet_service.cc:1467] Processing CreateTablet for tablet ba65d8d08aab4c9da9d10cb214286943 (DEFAULT_TABLE table=test-table [id=f76a42b8c7ec4618b095c00c4fb8a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.974954 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65d8d08aab4c9da9d10cb214286943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.974977 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65d8d08aab4c9da9d10cb214286943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.975610 23514 tablet_service.cc:1467] Processing CreateTablet for tablet ba65d8d08aab4c9da9d10cb214286943 (DEFAULT_TABLE table=test-table [id=f76a42b8c7ec4618b095c00c4fb8a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:28:44.975834 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba65d8d08aab4c9da9d10cb214286943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:44.977089 23749 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: Bootstrap starting.
I20260927 16:28:44.977790 23749 tablet_bootstrap.cc:654] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.977850 23748 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Bootstrap starting.
I20260927 16:28:44.978291 23750 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Bootstrap starting.
I20260927 16:28:44.978615 23748 tablet_bootstrap.cc:654] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.979072 23750 tablet_bootstrap.cc:654] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:44.979852 23748 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: No bootstrap required, opened a new log
I20260927 16:28:44.979938 23748 ts_tablet_manager.cc:1397] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:44.980149 23750 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: No bootstrap required, opened a new log
I20260927 16:28:44.980216 23750 ts_tablet_manager.cc:1397] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:28:44.980394 23749 tablet_bootstrap.cc:492] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: No bootstrap required, opened a new log
I20260927 16:28:44.980347 23748 raft_consensus.cc:348] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.980449 23748 raft_consensus.cc:374] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.980463 23749 ts_tablet_manager.cc:1397] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 16:28:44.980485 23748 raft_consensus.cc:729] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4d414f14454310828190f716607b99, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.980559 23750 raft_consensus.cc:348] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.980651 23750 raft_consensus.cc:374] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.980620 23748 consensus_queue.cc:260] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.980681 23750 raft_consensus.cc:729] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.980786 23750 consensus_queue.cc:260] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.980882 23748 ts_tablet_manager.cc:1428] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:44.980851 23749 raft_consensus.cc:348] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.980948 23749 raft_consensus.cc:374] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:44.980981 23749 raft_consensus.cc:729] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dac72e7e33f45ea992cdf5cb59c90f5, State: Initialized, Role: FOLLOWER
I20260927 16:28:44.981012 23750 ts_tablet_manager.cc:1428] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 16:28:44.981109 23749 consensus_queue.cc:260] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:44.981360 23749 ts_tablet_manager.cc:1428] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:28:45.052047 23756 raft_consensus.cc:482] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:28:45.052171 23756 raft_consensus.cc:504] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:45.052608 23756 leader_election.cc:288] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:45.053004 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65d8d08aab4c9da9d10cb214286943" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99" is_pre_election: true
I20260927 16:28:45.053022 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65d8d08aab4c9da9d10cb214286943" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" is_pre_election: true
I20260927 16:28:45.053174 23533 raft_consensus.cc:2454] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:45.053189 23394 raft_consensus.cc:2454] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 0.
I20260927 16:28:45.053560 23605 leader_election.cc:302] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:45.053748 23756 raft_consensus.cc:2781] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:28:45.053826 23756 raft_consensus.cc:482] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:28:45.053880 23756 raft_consensus.cc:3037] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:45.054807 23756 raft_consensus.cc:504] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:45.055159 23756 leader_election.cc:288] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Requested vote from peers 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865), 4d4d414f14454310828190f716607b99 (127.15.93.130:40249)
I20260927 16:28:45.055480 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65d8d08aab4c9da9d10cb214286943" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5"
I20260927 16:28:45.055595 23394 raft_consensus.cc:3037] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:45.055609 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba65d8d08aab4c9da9d10cb214286943" candidate_uuid: "95cb20409fc942b694c73ea5b96de915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d4d414f14454310828190f716607b99"
I20260927 16:28:45.055749 23533 raft_consensus.cc:3037] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:45.056761 23394 raft_consensus.cc:2454] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:45.056950 23533 raft_consensus.cc:2454] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95cb20409fc942b694c73ea5b96de915 in term 1.
I20260927 16:28:45.057052 23605 leader_election.cc:302] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95cb20409fc942b694c73ea5b96de915, 9dac72e7e33f45ea992cdf5cb59c90f5; no voters:
I20260927 16:28:45.057219 23756 raft_consensus.cc:2781] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:45.057554 23756 raft_consensus.cc:686] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 1 LEADER]: Becoming Leader. State: Replica: 95cb20409fc942b694c73ea5b96de915, State: Running, Role: LEADER
I20260927 16:28:45.057699 23756 consensus_queue.cc:237] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } }
I20260927 16:28:45.059185 23235 catalog_manager.cc:5495] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95cb20409fc942b694c73ea5b96de915 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "95cb20409fc942b694c73ea5b96de915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95cb20409fc942b694c73ea5b96de915" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 41835 } health_report { overall_health: HEALTHY } } }
I20260927 16:28:45.092675 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:45.089528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:39493 (local address 127.15.93.131:41835)
0927 16:28:45.089671 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:45.089676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:45.089690 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:45.089825 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:45.089829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:45.089877 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:45.089941 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:28:45.090230 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:45.090735 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:45.091711 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:45.091919 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:45.092024 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:45.092264 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:28:45.092325 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:28:45.092400 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:45.092486 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 16:28:45.092524 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":58}
I20260927 16:28:45.092700 23837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:45.089333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:41835 (local address 127.15.93.131:39493)
0927 16:28:45.089544 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:45.089577 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:45.089717 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:45.090063 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:45.090065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:45.090070 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 16:28:45.090134 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:45.090138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:45.090839 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:45.090845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:45.091594 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:45.091604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:45.092021 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:45.092025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:45.092118 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:45.092141 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 16:28:45.092401 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 16:28:45.092409 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:28:45.092470 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 16:28:45.092546 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260927 16:28:45.098662 23533 raft_consensus.cc:1261] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:45.098789 23394 raft_consensus.cc:1261] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Refusing update from remote peer 95cb20409fc942b694c73ea5b96de915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:28:45.099107 23803 consensus_queue.cc:1035] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dac72e7e33f45ea992cdf5cb59c90f5" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:45.099318 23793 consensus_queue.cc:1035] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d4d414f14454310828190f716607b99" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 40249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:28:45.133862 23236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:45.134001 23236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:28:45.135879 23236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e2907a06107478e9bdea634a5dbad4e: Sending DeleteTablet for 3 replicas of tablet ba65d8d08aab4c9da9d10cb214286943
I20260927 16:28:45.136417 23514 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65d8d08aab4c9da9d10cb214286943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:45 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260927 16:28:45.136463 23654 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65d8d08aab4c9da9d10cb214286943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:45 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35720
I20260927 16:28:45.136631 23772 tablet_replica.cc:330] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: stopping tablet replica
I20260927 16:28:45.136651 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet ba65d8d08aab4c9da9d10cb214286943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:28:45 UTC) from {username='oryx', principal='oryx/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 16:28:45.136755 23772 raft_consensus.cc:2229] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:28:45.136825 23773 tablet_replica.cc:330] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5: stopping tablet replica
I20260927 16:28:45.136838 23771 tablet_replica.cc:330] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: stopping tablet replica
I20260927 16:28:45.136937 23771 raft_consensus.cc:2229] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:45.136935 23773 raft_consensus.cc:2229] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:28:45.136937 23772 raft_consensus.cc:2258] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:45.137035 23771 raft_consensus.cc:2258] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:45.137035 23773 raft_consensus.cc:2258] T ba65d8d08aab4c9da9d10cb214286943 P 9dac72e7e33f45ea992cdf5cb59c90f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:28:45.137115 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23309
I20260927 16:28:45.138820 23772 ts_tablet_manager.cc:1905] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:45.139099 23771 ts_tablet_manager.cc:1905] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:28:45.141175 23772 ts_tablet_manager.cc:1918] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:45.141269 23772 log.cc:1196] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/ba65d8d08aab4c9da9d10cb214286943
I20260927 16:28:45.141508 23772 ts_tablet_manager.cc:1939] T ba65d8d08aab4c9da9d10cb214286943 P 95cb20409fc942b694c73ea5b96de915: Deleting consensus metadata
I20260927 16:28:45.142096 23223 catalog_manager.cc:4841] TS 95cb20409fc942b694c73ea5b96de915 (127.15.93.131:41835): tablet ba65d8d08aab4c9da9d10cb214286943 (table test-table [id=f76a42b8c7ec4618b095c00c4fb8a473]) successfully deleted
I20260927 16:28:45.142236 23771 ts_tablet_manager.cc:1918] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 16:28:45.142321 23771 log.cc:1196] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/ba65d8d08aab4c9da9d10cb214286943
I20260927 16:28:45.142568 23771 ts_tablet_manager.cc:1939] T ba65d8d08aab4c9da9d10cb214286943 P 4d4d414f14454310828190f716607b99: Deleting consensus metadata
I20260927 16:28:45.143240 23224 catalog_manager.cc:4841] TS 4d4d414f14454310828190f716607b99 (127.15.93.130:40249): tablet ba65d8d08aab4c9da9d10cb214286943 (table test-table [id=f76a42b8c7ec4618b095c00c4fb8a473]) successfully deleted
W20260927 16:28:45.145648 23221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:32865 (error 108)
I20260927 16:28:45.145866 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23449
I20260927 16:28:45.151799 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:45.146652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32865 (local address 127.0.0.1:58014)
0927 16:28:45.151579 (+ 4927us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:45.151672 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4825,"thread_start_us":56,"threads_started":1}
W20260927 16:28:45.151954 23221 catalog_manager.cc:4584] TS 9dac72e7e33f45ea992cdf5cb59c90f5 (127.15.93.129:32865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba65d8d08aab4c9da9d10cb214286943: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32865: connect: Connection refused (error 111)
I20260927 16:28:45.152870 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23589
I20260927 16:28:45.159930 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23205
2026-09-27T16:28:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3404 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:45 dist-test-slave-grzf krb5kdc[23845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:45 dist-test-slave-grzf krb5kdc[23845](info): set up 2 sockets
Sep 27 16:28:45 dist-test-slave-grzf krb5kdc[23845](info): commencing operation
krb5kdc: starting...
W20260927 16:28:46.208199 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:46 dist-test-slave-grzf krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:46.321396 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40293
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:46.427471 23858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:46.427727 23858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:46.427796 23858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:46.431805 23858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:46.431857 23858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:46.431876 23858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:46.431893 23858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:46.431944 23858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:46.437050 23858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:46.438035 23858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:46.439013 23858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:46.447197 23866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:46.447197 23864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:46.447319 23858 server_base.cc:1034] running on GCE node
W20260927 16:28:46.447224 23863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:46.448014 23858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:46.449074 23858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:46.450270 23858 hybrid_clock.cc:648] HybridClock initialized: now 1790526526450260 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T16:28:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1297 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:28:46 dist-test-slave-grzf krb5kdc[23871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:28:46 dist-test-slave-grzf krb5kdc[23871](info): set up 2 sockets
Sep 27 16:28:46 dist-test-slave-grzf krb5kdc[23871](info): commencing operation
krb5kdc: starting...
W20260927 16:28:47.496309 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:28:47 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:28:47.613828 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44319
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:47.719584 23884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:47.719826 23884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:47.719864 23884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:47.723670 23884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:47.723714 23884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:47.723733 23884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:47.723749 23884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:47.723763 23884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:47.728686 23884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:47.729564 23884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:47.730446 23884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:47.738250 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:47.738262 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:47.738397 23884 server_base.cc:1034] running on GCE node
W20260927 16:28:47.738554 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:47.739089 23884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:47.740176 23884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:47.741385 23884 hybrid_clock.cc:648] HybridClock initialized: now 1790526527741369 us; error 35 us; skew 500 ppm
Sep 27 16:28:47 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:47.747618 23884 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:47.748530 23884 webserver.cc:460] Webserver started at http://127.15.93.190:37101/ using document root <none> and password file <none>
I20260927 16:28:47.749090 23884 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:47.749189 23884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:47.749411 23884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:47.751430 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "0d3adbbb252441d086985a79a6db4881"
format_stamp: "Formatted at 2026-09-27 16:28:47 on dist-test-slave-grzf"
server_key: "e0422b2e713e79d6e7407b78f839b67d"
server_key_iv: "0b512869930dd9f214876f9d11c9946b"
server_key_version: "encryptionkey@0"
I20260927 16:28:47.751988 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "0d3adbbb252441d086985a79a6db4881"
format_stamp: "Formatted at 2026-09-27 16:28:47 on dist-test-slave-grzf"
server_key: "e0422b2e713e79d6e7407b78f839b67d"
server_key_iv: "0b512869930dd9f214876f9d11c9946b"
server_key_version: "encryptionkey@0"
I20260927 16:28:47.755532 23884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:28:47.757737 23899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:47.758893 23884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:47.759053 23884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0d3adbbb252441d086985a79a6db4881"
format_stamp: "Formatted at 2026-09-27 16:28:47 on dist-test-slave-grzf"
server_key: "e0422b2e713e79d6e7407b78f839b67d"
server_key_iv: "0b512869930dd9f214876f9d11c9946b"
server_key_version: "encryptionkey@0"
I20260927 16:28:47.759166 23884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:47.801486 23884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:47.804630 23884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:47.804867 23884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:47.812260 23884 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:44319
I20260927 16:28:47.812285 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:44319 every 8 connection(s)
I20260927 16:28:47.813202 23884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:47.815876 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:28:47.819516 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23884
I20260927 16:28:47.819660 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:28:47.819967 15734 external_mini_cluster.cc:1428] Setting key ca6801045b1453fccd6a5152d2139c57
I20260927 16:28:47.821698 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Bootstrap starting.
I20260927 16:28:47.823851 23952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Neither blocks nor log segments found. Creating new log.
I20260927 16:28:47.824610 23952 log.cc:824] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:47.826356 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: No bootstrap required, opened a new log
Sep 27 16:28:47 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:47.831065 23952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:47.831338 23952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:28:47.831419 23952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3adbbb252441d086985a79a6db4881, State: Initialized, Role: FOLLOWER
I20260927 16:28:47.831907 23952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:47.832069 23952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:47.832149 23952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:47.832252 23952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:28:47.833227 23952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:47.833602 23952 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3adbbb252441d086985a79a6db4881; no voters:
I20260927 16:28:47.833904 23952 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:28:47.834003 23957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:28:47.834208 23957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3adbbb252441d086985a79a6db4881, State: Running, Role: LEADER
I20260927 16:28:47.834501 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:47.821365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.15.93.190:44319)
0927 16:28:47.821797 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:47.821817 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:47.821864 (+ 47us) server_negotiation.cc:407] Connection header received
0927 16:28:47.822475 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:47.822491 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:47.822802 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:47.823159 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:47.824060 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:47.825125 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:47.825968 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:47.826424 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:47.826609 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:47.829304 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:47.829312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:47.829332 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:47.829379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:47.831349 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:47.831872 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:47.831878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:47.831883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:47.832023 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:47.832370 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:47.832374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:47.832376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:47.832775 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:47.832933 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:47.833234 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 16:28:47.833521 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260927 16:28:47.834510 23957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:47.835171 23952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:47.835695 23957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3adbbb252441d086985a79a6db4881. Latest consensus state: current_term: 1 leader_uuid: "0d3adbbb252441d086985a79a6db4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } } }
I20260927 16:28:47.835809 23957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:47.836190 23958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3adbbb252441d086985a79a6db4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } } }
I20260927 16:28:47.836273 23958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:47.837162 23962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:47.839911 23962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:47.845777 23962 catalog_manager.cc:1347] Generated new cluster ID: f391497e94594ff186134bfcf3c78896
I20260927 16:28:47.845906 23962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:47.865260 23962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:28:47.866179 23962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:47.876435 23962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Generated new TSK 0
I20260927 16:28:47.877130 23962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:28:47.909987 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:48.018592 23978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:48.018846 23978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:48.018886 23978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:48.022816 23978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:48.022872 23978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:48.022967 23978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:48.028364 23978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:48.029376 23978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:48.030385 23978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:48.039646 23983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.039748 23978 server_base.cc:1034] running on GCE node
W20260927 16:28:48.039706 23984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.039652 23986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.040352 23978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:48.040978 23978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:48.042172 23978 hybrid_clock.cc:648] HybridClock initialized: now 1790526528042149 us; error 35 us; skew 500 ppm
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:48.044898 23978 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:28:48.045796 23978 webserver.cc:460] Webserver started at http://127.15.93.129:34675/ using document root <none> and password file <none>
I20260927 16:28:48.046355 23978 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:48.046443 23978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:48.046670 23978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:48.048704 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "71ae07cb1c71443c9124e4449bd18a1e"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "d37b7fc6bca14a106cbd3bad741b4e34"
server_key_iv: "7ff33e8b631479d372a8d3b5ff223ea7"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.049275 23978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "71ae07cb1c71443c9124e4449bd18a1e"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "d37b7fc6bca14a106cbd3bad741b4e34"
server_key_iv: "7ff33e8b631479d372a8d3b5ff223ea7"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.052879 23978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:28:48.055138 23993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.056341 23978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:28:48.056471 23978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "71ae07cb1c71443c9124e4449bd18a1e"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "d37b7fc6bca14a106cbd3bad741b4e34"
server_key_iv: "7ff33e8b631479d372a8d3b5ff223ea7"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.056577 23978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:48.071092 23978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:48.074016 23978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:48.074230 23978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:48.074829 23978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:48.075721 23978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:48.075786 23978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.075850 23978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:48.075901 23978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.085278 23978 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:33585
I20260927 16:28:48.085295 24106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:33585 every 8 connection(s)
I20260927 16:28:48.086220 23978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:48.096074 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 23978
I20260927 16:28:48.096231 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:28:48.096618 15734 external_mini_cluster.cc:1428] Setting key f95155ec968b603a469711875e31641e
I20260927 16:28:48.099304 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.087981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:48953 (local address 127.15.93.190:44319)
0927 16:28:48.088144 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.088148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.089110 (+ 962us) server_negotiation.cc:407] Connection header received
0927 16:28:48.089925 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.089927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.089974 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.090050 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.091155 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.091761 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.092842 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.093059 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.093133 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.095467 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:48.095470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:48.095472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:48.095495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:48.097244 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.097733 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.097737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.097739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.097828 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.098227 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.098230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.098231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.098398 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:48.098475 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:48.099084 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 16:28:48.099188 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:28:48.100358 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.088348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:48953)
0927 16:28:48.088972 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.089011 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.089751 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.090215 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.090222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.090563 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.091028 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.091041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.091872 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.091874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.092743 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.092752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.093191 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.093194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.093329 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.093862 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.093878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.095273 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:48.097348 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.097354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.097371 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.097628 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.097930 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.097932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.097933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.098121 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.098511 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:48.098521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:48.098836 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 16:28:48.099049 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:28:48.099442 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":107,"threads_started":1}
I20260927 16:28:48.101569 24107 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44319
I20260927 16:28:48.101818 24107 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:48.102265 24107 heartbeater.cc:511] Master 127.15.93.190:44319 requested a full tablet report, sending...
I20260927 16:28:48.103821 23916 ts_manager.cc:194] Registered new tserver with Master: 71ae07cb1c71443c9124e4449bd18a1e (127.15.93.129:33585)
I20260927 16:28:48.104883 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:48953
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:28:48.122598 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:48.235272 24112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:48.235526 24112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:48.235596 24112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:48.239739 24112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:48.239790 24112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:48.239845 24112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:48.244947 24112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:48.245939 24112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:48.246919 24112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:48.256160 24120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.256173 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.256173 24117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.256384 24112 server_base.cc:1034] running on GCE node
I20260927 16:28:48.256915 24112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:48.257534 24112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:48.258692 24112 hybrid_clock.cc:648] HybridClock initialized: now 1790526528258656 us; error 47 us; skew 500 ppm
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:48.261552 24112 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:28:48.262490 24112 webserver.cc:460] Webserver started at http://127.15.93.130:38651/ using document root <none> and password file <none>
I20260927 16:28:48.263015 24112 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:48.263072 24112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:48.263350 24112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:48.265341 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "fd0b8cc48245484c8ee632ea4fe2c228"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "3361b304b0eee03871603f8656c21c6f"
server_key_iv: "59b5150ffa34ccbccf5f930318d3a2bb"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.265837 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "fd0b8cc48245484c8ee632ea4fe2c228"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "3361b304b0eee03871603f8656c21c6f"
server_key_iv: "59b5150ffa34ccbccf5f930318d3a2bb"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.269382 24112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:28:48.271627 24127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.272769 24112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:28:48.272857 24112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "fd0b8cc48245484c8ee632ea4fe2c228"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "3361b304b0eee03871603f8656c21c6f"
server_key_iv: "59b5150ffa34ccbccf5f930318d3a2bb"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.272936 24112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:48.282817 24112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:48.285409 24112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:48.285544 24112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:48.286016 24112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:48.286715 24112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:48.286759 24112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.286795 24112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:48.286816 24112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.295293 24112 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:39705
I20260927 16:28:48.295315 24240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:39705 every 8 connection(s)
I20260927 16:28:48.296186 24112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:48.301143 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24112
I20260927 16:28:48.301290 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:28:48.301590 15734 external_mini_cluster.cc:1428] Setting key 194b992e9ac4ca125b4a15ac7ce83645
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:48.310231 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.297908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41885 (local address 127.15.93.190:44319)
0927 16:28:48.298042 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.298046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.298989 (+ 943us) server_negotiation.cc:407] Connection header received
0927 16:28:48.299856 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.299859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.299905 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.299978 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.301106 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.301906 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.303356 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.303782 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.303904 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.306551 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:48.306555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:48.306557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:48.306580 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:48.308166 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.308709 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.308715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.308717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.308800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.309147 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.309150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.309152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.309333 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:48.309440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:48.309919 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 16:28:48.310050 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:28:48.311036 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.298249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:41885)
0927 16:28:48.298846 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.298885 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.299671 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.300128 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.300134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.300464 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.300961 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.300974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.302039 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.302043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.303242 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.303252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.303940 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.303944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.304132 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.304738 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.304765 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.306384 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:48.308304 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.308310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.308324 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.308584 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.308901 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.308904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.308906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.309042 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.309438 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:48.309445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:48.309693 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 16:28:48.309880 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 16:28:48.310192 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":103,"threads_started":1}
I20260927 16:28:48.312227 24241 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44319
I20260927 16:28:48.312484 24241 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:28:48.312981 24241 heartbeater.cc:511] Master 127.15.93.190:44319 requested a full tablet report, sending...
I20260927 16:28:48.313982 23916 ts_manager.cc:194] Registered new tserver with Master: fd0b8cc48245484c8ee632ea4fe2c228 (127.15.93.130:39705)
I20260927 16:28:48.314797 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:41885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:28:48.326139 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:48.432816 24246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:48.433058 24246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:48.433099 24246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:48.437211 24246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:48.437263 24246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:48.437319 24246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:48.442406 24246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:48.443307 24246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:48.444305 24246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:48.453496 24252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.453526 24251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.453526 24254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.453644 24246 server_base.cc:1034] running on GCE node
I20260927 16:28:48.454109 24246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:48.454712 24246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:48.455890 24246 hybrid_clock.cc:648] HybridClock initialized: now 1790526528455874 us; error 41 us; skew 500 ppm
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:48.458591 24246 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:28:48.459502 24246 webserver.cc:460] Webserver started at http://127.15.93.131:35795/ using document root <none> and password file <none>
I20260927 16:28:48.460089 24246 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:48.460186 24246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:48.460393 24246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:28:48.462433 24246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "930b5d93fa7a46fd98d8c4a6b3e014fd"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "703870fc00e267f0c4ceac6f7a3c8428"
server_key_iv: "cb13fea2dd2ffd15a7c70665c839bd2f"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.462986 24246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "930b5d93fa7a46fd98d8c4a6b3e014fd"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "703870fc00e267f0c4ceac6f7a3c8428"
server_key_iv: "cb13fea2dd2ffd15a7c70665c839bd2f"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.466508 24246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 16:28:48.468839 24261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.470005 24246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:48.470139 24246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "930b5d93fa7a46fd98d8c4a6b3e014fd"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "703870fc00e267f0c4ceac6f7a3c8428"
server_key_iv: "cb13fea2dd2ffd15a7c70665c839bd2f"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.470245 24246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:48.486402 24246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:48.489537 24246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:48.489732 24246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:48.490283 24246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:48.491043 24246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:48.491110 24246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.491187 24246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:48.491231 24246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.499475 24246 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:42029
I20260927 16:28:48.499513 24374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:42029 every 8 connection(s)
I20260927 16:28:48.500280 24246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:48.501497 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24246
I20260927 16:28:48.501638 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:28:48.501962 15734 external_mini_cluster.cc:1428] Setting key 5a125ad62ac84ddaeee486455016ae02
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:28:48.512575 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.501990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:52527 (local address 127.15.93.190:44319)
0927 16:28:48.502133 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.502139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.503309 (+ 1170us) server_negotiation.cc:407] Connection header received
0927 16:28:48.504346 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.504349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.504409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.504497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.505586 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.506098 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.507147 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.507346 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.507496 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.509720 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:48.509723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:48.509725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:48.509748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:48.510896 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.511326 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.511329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.511330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.511453 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:48.511746 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:48.511748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:48.511749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:48.511907 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:48.511979 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:48.512373 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 16:28:48.512461 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:28:48.513316 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.502381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:52527)
0927 16:28:48.503132 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.503182 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.504138 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.504677 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.504684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.505011 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.505469 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.505481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.506204 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.506207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.507047 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.507056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.507527 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.507530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.507664 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.508141 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.508156 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.509562 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:28:48.510990 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.510994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.511003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.511245 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.511529 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:28:48.511531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:28:48.511532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:28:48.511667 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:28:48.511988 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:28:48.511992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:28:48.512187 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 16:28:48.512329 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 16:28:48.512548 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":137,"threads_started":1}
I20260927 16:28:48.514338 24375 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44319
I20260927 16:28:48.514580 24375 heartbeater.cc:464] Registering TS with master...
I20260927 16:28:48.515035 24375 heartbeater.cc:511] Master 127.15.93.190:44319 requested a full tablet report, sending...
I20260927 16:28:48.516007 23916 ts_manager.cc:194] Registered new tserver with Master: 930b5d93fa7a46fd98d8c4a6b3e014fd (127.15.93.131:42029)
I20260927 16:28:48.516534 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:52527
I20260927 16:28:48.527164 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
Entry for principal oryx/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab.
WARNING: no policy specified for oryx/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
Entry for principal oryx/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab.
WARNING: no policy specified for oryx/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
Entry for principal oryx/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab.
I20260927 16:28:48.591224 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23978
I20260927 16:28:48.596803 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24112
I20260927 16:28:48.601815 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24246
I20260927 16:28:48.607009 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 23884
I20260927 16:28:48.613241 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44319
--webserver_interface=127.15.93.190
--webserver_port=37101
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:28:48.717425 24384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:48.717653 24384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:48.717689 24384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:48.721676 24384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:28:48.721722 24384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:48.721740 24384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:48.721756 24384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:28:48.721771 24384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:28:48.726733 24384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:48.727670 24384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:48.728574 24384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:48.736395 24392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.736402 24389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.736402 24390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.736548 24384 server_base.cc:1034] running on GCE node
I20260927 16:28:48.737185 24384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:48.738157 24384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:48.739311 24384 hybrid_clock.cc:648] HybridClock initialized: now 1790526528739299 us; error 24 us; skew 500 ppm
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:48.741787 24384 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:28:48.742604 24384 webserver.cc:460] Webserver started at http://127.15.93.190:37101/ using document root <none> and password file <none>
I20260927 16:28:48.743113 24384 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:48.743216 24384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:48.746628 24384 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:48.748570 24399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.749637 24384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:28:48.749774 24384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0d3adbbb252441d086985a79a6db4881"
format_stamp: "Formatted at 2026-09-27 16:28:47 on dist-test-slave-grzf"
server_key: "e0422b2e713e79d6e7407b78f839b67d"
server_key_iv: "0b512869930dd9f214876f9d11c9946b"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.750247 24384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:48.761349 24384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:48.763727 24384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:48.763906 24384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:48.770288 24384 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:44319
I20260927 16:28:48.770316 24451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:44319 every 8 connection(s)
I20260927 16:28:48.771234 24384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:28:48.774932 24452 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:28:48.776486 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Bootstrap starting.
I20260927 16:28:48.778738 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24384
I20260927 16:28:48.779276 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:33585
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:44319
--webserver_port=34675
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:28:48.792887 24452 log.cc:824] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Log is configured to *not* fsync() on all Append() calls
I20260927 16:28:48.798919 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:28:48.799264 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Bootstrap complete.
I20260927 16:28:48.803311 24452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:48.803732 24452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3adbbb252441d086985a79a6db4881, State: Initialized, Role: FOLLOWER
I20260927 16:28:48.804188 24452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:48.804328 24452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:28:48.804394 24452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:28:48.804512 24452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:28:48.806345 24452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:48.806655 24452 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3adbbb252441d086985a79a6db4881; no voters:
I20260927 16:28:48.806952 24452 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:28:48.807055 24456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:28:48.807293 24456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [term 2 LEADER]: Becoming Leader. State: Replica: 0d3adbbb252441d086985a79a6db4881, State: Running, Role: LEADER
I20260927 16:28:48.807662 24456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } }
I20260927 16:28:48.808130 24452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:28:48.809019 24457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d3adbbb252441d086985a79a6db4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } } }
I20260927 16:28:48.809151 24457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:48.809494 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3adbbb252441d086985a79a6db4881. Latest consensus state: current_term: 2 leader_uuid: "0d3adbbb252441d086985a79a6db4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3adbbb252441d086985a79a6db4881" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44319 } } }
I20260927 16:28:48.809644 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881 [sys.catalog]: This master's current role is: LEADER
I20260927 16:28:48.809924 24465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:28:48.812912 24465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:28:48.813568 24465 catalog_manager.cc:1259] Loaded cluster ID: f391497e94594ff186134bfcf3c78896
I20260927 16:28:48.813632 24465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:28:48.815937 24465 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:28:48.816545 24465 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d3adbbb252441d086985a79a6db4881: Loaded TSK: 0
I20260927 16:28:48.817663 24465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:28:48.890692 24454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:48.890929 24454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:48.890970 24454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:48.894922 24454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:48.894976 24454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:48.895033 24454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:28:48.900220 24454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.129.keytab
--principal=oryx/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:33585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=34675
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:48.901190 24454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:48.902135 24454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:48.911476 24481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.911479 24479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:48.911476 24478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:48.911937 24454 server_base.cc:1034] running on GCE node
I20260927 16:28:48.912348 24454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:48.913000 24454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:48.914170 24454 hybrid_clock.cc:648] HybridClock initialized: now 1790526528914146 us; error 45 us; skew 500 ppm
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526528, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:48.916981 24454 init.cc:377] Logged in from keytab as oryx/127.15.93.129@KRBTEST.COM (short username oryx)
I20260927 16:28:48.917819 24454 webserver.cc:460] Webserver started at http://127.15.93.129:34675/ using document root <none> and password file <none>
I20260927 16:28:48.918455 24454 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:48.918565 24454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:48.922387 24454 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:48.924477 24488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.925717 24454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:48.925843 24454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "71ae07cb1c71443c9124e4449bd18a1e"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "d37b7fc6bca14a106cbd3bad741b4e34"
server_key_iv: "7ff33e8b631479d372a8d3b5ff223ea7"
server_key_version: "encryptionkey@0"
I20260927 16:28:48.926344 24454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:48.942448 24454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:48.945376 24454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:48.945576 24454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:48.946128 24454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:48.946959 24454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:48.947026 24454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.947093 24454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:48.947150 24454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:48.956120 24454 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:33585
I20260927 16:28:48.956188 24601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:33585 every 8 connection(s)
I20260927 16:28:48.957084 24454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:28:48.965739 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24454
I20260927 16:28:48.966192 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:39705
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:44319
--webserver_port=38651
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:48.969818 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.959102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:60743)
0927 16:28:48.959749 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.959795 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.960538 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.961830 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.961837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.962188 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.962634 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.962647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.963900 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.963903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.964770 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.964779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.965578 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.965581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.965711 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.966363 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.966384 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.969106 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":125,"threads_started":1}
W20260927 16:28:48.970289 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:48.958950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:60743 (local address 127.15.93.190:44319)
0927 16:28:48.959362 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.959420 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.959900 (+ 480us) server_negotiation.cc:407] Connection header received
0927 16:28:48.961176 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.961187 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.961456 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.961727 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.962802 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.963766 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.964881 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.965386 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.965586 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.969406 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:60743: BlockingRecv error: recv got EOF from 127.15.93.129:60743 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
W20260927 16:28:48.970417 24602 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44319 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.93.190:44319: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:48.976706 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.971114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:49583)
0927 16:28:48.971232 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.971248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.971325 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.971618 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.971620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.971745 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.971829 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.971834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.972943 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.972945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.973460 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.973467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.974120 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.974123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.974194 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.974710 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.974722 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.976605 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 16:28:48.976951 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:48.971196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:49583 (local address 127.15.93.190:44319)
0927 16:28:48.971337 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.971343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.971359 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:48.971443 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.971449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.971505 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.971607 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.971940 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.972826 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.973570 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.973985 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.974112 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.976788 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:49583: BlockingRecv error: recv got EOF from 127.15.93.129:49583 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:48 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:48.982401 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:48.977346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:43035)
0927 16:28:48.977432 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:48.977448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:48.977519 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:48.977762 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:48.977764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:48.977876 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:48.977938 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.977942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.978907 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.978909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.979514 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:48.979523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.980069 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.980089 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:48.980179 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.980646 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:48.980657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:48.982316 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 16:28:48.982565 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:48.977421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:43035 (local address 127.15.93.190:44319)
0927 16:28:48.977547 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:48.977551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:48.977566 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:48.977608 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:48.977612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:48.977663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:48.977762 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:48.978034 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.978807 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.979612 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:48.979964 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:48.980055 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:48.982461 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:43035: BlockingRecv error: recv got EOF from 127.15.93.129:43035 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:28:48.982774 24602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:28:49.076607 24606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:49.076864 24606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:49.076941 24606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:49.080881 24606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:49.080929 24606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:49.081036 24606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:28:49.086088 24606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.130.keytab
--principal=oryx/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:39705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=38651
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:49.087069 24606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:49.088052 24606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:49.097820 24611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:49.097901 24606 server_base.cc:1034] running on GCE node
W20260927 16:28:49.097833 24614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:49.097820 24612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:49.098363 24606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:49.098994 24606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:49.100169 24606 hybrid_clock.cc:648] HybridClock initialized: now 1790526529100156 us; error 25 us; skew 500 ppm
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526529, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:49.102730 24606 init.cc:377] Logged in from keytab as oryx/127.15.93.130@KRBTEST.COM (short username oryx)
I20260927 16:28:49.103574 24606 webserver.cc:460] Webserver started at http://127.15.93.130:38651/ using document root <none> and password file <none>
I20260927 16:28:49.104102 24606 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:49.104211 24606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:49.107926 24606 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 16:28:49.110015 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.111143 24606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:49.111274 24606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "fd0b8cc48245484c8ee632ea4fe2c228"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "3361b304b0eee03871603f8656c21c6f"
server_key_iv: "59b5150ffa34ccbccf5f930318d3a2bb"
server_key_version: "encryptionkey@0"
I20260927 16:28:49.111846 24606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:49.130318 24606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:49.132985 24606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:49.133174 24606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:49.133718 24606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:49.134483 24606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:49.134559 24606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.134639 24606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:49.134694 24606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.143965 24606 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:39705
I20260927 16:28:49.144007 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:39705 every 8 connection(s)
I20260927 16:28:49.145294 24606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:28:49.152850 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24606
I20260927 16:28:49.153321 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:42029
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:44319
--webserver_port=35795
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:28:49.159538 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.147244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:33305 (local address 127.15.93.190:44319)
0927 16:28:49.147409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.147414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.148795 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 16:28:49.149925 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.149928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.150007 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.150106 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.151679 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.152522 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.154578 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.154966 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.155055 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.159367 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:33305: BlockingRecv error: recv got EOF from 127.15.93.130:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:28:49.160250 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.147826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:33305)
0927 16:28:49.148628 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.148697 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.149717 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.150280 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.150289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.150796 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.151470 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.151494 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.152710 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.152717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.154405 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.154419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.155300 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.155304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.155548 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.156204 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.156226 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.159169 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":124,"threads_started":1}
W20260927 16:28:49.160743 24735 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44319 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.93.190:44319: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.167876 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.161191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:47423)
0927 16:28:49.161312 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.161330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.161418 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.161739 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.161742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.161920 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.162057 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.162067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.162830 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.162835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.163850 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.163864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.164496 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.164503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.164633 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.165647 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.165667 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.167760 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 16:28:49.168121 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.161272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:47423 (local address 127.15.93.190:44319)
0927 16:28:49.161378 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.161381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.161392 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:49.161542 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.161544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.161590 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.161701 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.162188 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.162721 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.164021 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.164358 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.164468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.167980 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:47423: BlockingRecv error: recv got EOF from 127.15.93.130:47423 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.174322 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.168709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:59383)
0927 16:28:49.168851 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.168863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.168925 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.169183 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.169186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.169334 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.169412 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.169418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.170418 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.170421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.171266 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.171275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.171973 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.171977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.172058 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.172527 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.172541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.174241 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 16:28:49.174489 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.168811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:59383 (local address 127.15.93.190:44319)
0927 16:28:49.168967 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.168971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.168985 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:49.169021 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.169025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.169068 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.169158 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.169519 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.170309 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.171437 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.171770 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.171839 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.174375 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:59383: BlockingRecv error: recv got EOF from 127.15.93.130:59383 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 16:28:49.174697 24735 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:28:49.267232 24738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:28:49.267534 24738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:28:49.267611 24738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:28:49.271581 24738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:28:49.271631 24738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:28:49.271716 24738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:28:49.276769 24738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/krb5kdc/oryx_127.15.93.131.keytab
--principal=oryx/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:42029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=35795
--tserver_master_addrs=127.15.93.190:44319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:28:49.277768 24738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:28:49.278733 24738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:28:49.287539 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:49.287530 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:28:49.287539 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:28:49.288007 24738 server_base.cc:1034] running on GCE node
I20260927 16:28:49.288362 24738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:28:49.288954 24738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:28:49.290118 24738 hybrid_clock.cc:648] HybridClock initialized: now 1790526529290103 us; error 25 us; skew 500 ppm
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526529, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:28:49.292775 24738 init.cc:377] Logged in from keytab as oryx/127.15.93.131@KRBTEST.COM (short username oryx)
I20260927 16:28:49.293658 24738 webserver.cc:460] Webserver started at http://127.15.93.131:35795/ using document root <none> and password file <none>
I20260927 16:28:49.294271 24738 fs_manager.cc:359] Metadata directory not provided
I20260927 16:28:49.294373 24738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:28:49.298082 24738 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 16:28:49.300101 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.301407 24738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:28:49.301535 24738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "930b5d93fa7a46fd98d8c4a6b3e014fd"
format_stamp: "Formatted at 2026-09-27 16:28:48 on dist-test-slave-grzf"
server_key: "703870fc00e267f0c4ceac6f7a3c8428"
server_key_iv: "cb13fea2dd2ffd15a7c70665c839bd2f"
server_key_version: "encryptionkey@0"
I20260927 16:28:49.302070 24738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:28:49.320647 24738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:28:49.323302 24738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:28:49.323519 24738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:28:49.324083 24738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:28:49.324865 24738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:28:49.324934 24738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.325003 24738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:28:49.325055 24738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:28:49.333930 24738 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:42029
I20260927 16:28:49.333951 24866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:42029 every 8 connection(s)
I20260927 16:28:49.334928 24738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:28:49.341377 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24738
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 16:28:49.345901 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.336676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:47029 (local address 127.15.93.190:44319)
0927 16:28:49.336806 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.336810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.337774 (+ 964us) server_negotiation.cc:407] Connection header received
0927 16:28:49.338581 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.338584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.338630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.338706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.339891 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.340402 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.341698 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.342028 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.342149 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.345776 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:47029: BlockingRecv error: recv got EOF from 127.15.93.131:47029 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:28:49.346519 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.337005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:47029)
0927 16:28:49.337633 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.337672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.338411 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.338861 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.338869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.339216 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.339766 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.339779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.340505 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.340508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.341585 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.341596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.342183 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.342187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.342383 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.343107 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.343123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.345636 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":111,"threads_started":1}
W20260927 16:28:49.347108 24867 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:44319 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.93.190:44319: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.351989 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.342401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.15.93.190:44319)
0927 16:28:49.342673 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.342676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.342687 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:49.342716 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.342718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.342751 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.342848 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.343824 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.344535 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.345726 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.346054 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.346190 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.347805 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:28:49.347812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:28:49.347831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:28:49.347868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:28:49.349827 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:49.350272 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:49.350283 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:49.350288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:49.350402 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:28:49.350682 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:28:49.350686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:28:49.350688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:28:49.351011 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:28:49.351224 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:28:49.351545 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 16:28:49.351808 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.352946 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.347600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:43379)
0927 16:28:49.347733 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.347748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.347812 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.348031 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.348033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.348207 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.348306 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.348311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.349151 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.349154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.349997 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.350012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.350427 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.350430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.350515 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.350944 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.350958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.352830 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 16:28:49.353082 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.347698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:43379 (local address 127.15.93.190:44319)
0927 16:28:49.347801 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.347804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.347815 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:49.347891 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.347892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.347924 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.347986 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.348442 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.349038 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.350110 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.350321 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.350422 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.352983 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:43379: BlockingRecv error: recv got EOF from 127.15.93.131:43379 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.358404 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.354020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:37351)
0927 16:28:49.354211 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.354226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.354295 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.354509 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.354512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.354660 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.354747 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.354754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.355437 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.355440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.355942 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.355949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.356326 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.356329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.356382 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.356816 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.356828 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.358321 (+ 1493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 16:28:49.358542 24604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.354126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:37351 (local address 127.15.93.190:44319)
0927 16:28:49.354302 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.354306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.354321 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:49.354358 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.354361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.354400 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.354472 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.354843 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.355311 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.356036 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.356235 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.356310 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.358433 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:37351: BlockingRecv error: recv got EOF from 127.15.93.131:37351 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 16:28:49.358633 24867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:49 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:49.988216 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:49.983327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:55661)
0927 16:28:49.983616 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:49.983631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:49.983713 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:49.983937 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:49.983939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:49.984108 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:49.984211 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.984216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.984900 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.984903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.985467 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:49.985474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.985872 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.985874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:49.985933 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.986395 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:49.986406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:49.988092 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":129,"threads_started":1}
W20260927 16:28:49.988399 24873 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:49.983424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:55661 (local address 127.15.93.190:44319)
0927 16:28:49.983662 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:49.983666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:49.983678 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:49.983790 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:49.983792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:49.983834 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:49.983908 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:49.984299 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.984816 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.985559 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:49.985780 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:49.985856 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:49.988230 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:55661: BlockingRecv error: recv got EOF from 127.15.93.129:55661 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":100,"threads_started":1}
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:50.181174 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:50.175354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:56469)
0927 16:28:50.175669 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:50.175683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:50.175780 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:50.175996 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:50.175998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:50.176195 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:50.176311 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.176319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.177275 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.177279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.178126 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.178137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.178555 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.178565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.178625 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.179094 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:50.179105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:50.181053 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
W20260927 16:28:50.181337 24873 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:50.175422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:56469 (local address 127.15.93.190:44319)
0927 16:28:50.175553 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:50.175557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:50.175756 (+ 199us) server_negotiation.cc:407] Connection header received
0927 16:28:50.175847 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:50.175849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:50.175893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:50.175969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:50.176422 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.177138 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.178238 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.178451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.178530 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.181187 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:56469: BlockingRecv error: recv got EOF from 127.15.93.130:56469 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:50.364831 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:50.359277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:38075)
0927 16:28:50.359525 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:50.359542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:50.359662 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:50.359990 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:50.359994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:50.360245 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:50.360372 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.360380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.361317 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.361320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.361880 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.361888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.362307 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.362310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.362368 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.362824 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:50.362834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:50.364659 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
W20260927 16:28:50.364874 24873 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:50.359427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38075 (local address 127.15.93.190:44319)
0927 16:28:50.359573 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:50.359577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:50.359591 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:28:50.359785 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:50.359790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:50.359858 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:50.359929 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:50.360503 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.361216 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.361999 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.362210 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.362289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.364772 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:38075: BlockingRecv error: recv got EOF from 127.15.93.131:38075 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:50 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:50.994107 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:50.988979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:47097)
0927 16:28:50.989254 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:50.989270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:50.989408 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:50.989674 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:50.989676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:50.989848 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:50.989939 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.989943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.990644 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.990647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.991159 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:50.991165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.991659 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.991662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:50.991722 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.992210 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:50.992231 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:50.993973 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
W20260927 16:28:50.994225 24877 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:50.989025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47097 (local address 127.15.93.190:44319)
0927 16:28:50.989288 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:50.989291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:50.989352 (+ 61us) server_negotiation.cc:407] Connection header received
0927 16:28:50.989493 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:50.989495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:50.989538 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:50.989639 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:50.990030 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.990553 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.991247 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:50.991563 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:50.991649 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:50.994098 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:47097: BlockingRecv error: recv got EOF from 127.15.93.129:47097 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:51.187178 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:51.181897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:41877)
0927 16:28:51.182163 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:51.182177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:51.182273 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:51.182482 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:51.182484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:51.182653 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:51.182744 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.182749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.183817 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.183820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.184436 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.184446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.184854 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.184856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.184915 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:51.185367 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:51.185378 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:51.187037 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
W20260927 16:28:51.187254 24877 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:51.181963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41877 (local address 127.15.93.190:44319)
0927 16:28:51.182093 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:51.182097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:51.182226 (+ 129us) server_negotiation.cc:407] Connection header received
0927 16:28:51.182337 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:51.182339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:51.182379 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:51.182447 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:51.182859 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.183707 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.184543 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.184760 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.184840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:51.187167 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:41877: BlockingRecv error: recv got EOF from 127.15.93.130:41877 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:51.370960 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:51.365617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:36849)
0927 16:28:51.365876 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:51.365891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:51.366010 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:51.366289 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:51.366292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:51.366501 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:51.366600 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.366605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.367361 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.367366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.367989 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.367999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.368456 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.368459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.368549 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:51.369020 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:51.369032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:51.370781 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260927 16:28:51.371001 24877 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:51.365809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:36849 (local address 127.15.93.190:44319)
0927 16:28:51.365944 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:51.365948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:51.365963 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:28:51.366119 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:51.366123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:51.366178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:51.366267 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:51.366721 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.367251 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.368114 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.368346 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.368454 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:51.370901 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:36849: BlockingRecv error: recv got EOF from 127.15.93.131:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:51 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:52.000501 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:51.994957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:33351)
0927 16:28:51.995362 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:51.995411 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:51.995547 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:51.995825 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:51.995827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:51.996036 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:51.996124 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.996130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.996904 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.996910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.997470 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:51.997477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.997946 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.997950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:51.998035 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:51.998541 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:51.998553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:52.000389 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
W20260927 16:28:52.000684 24881 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:51.995024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:33351 (local address 127.15.93.190:44319)
0927 16:28:51.995300 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:51.995304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:51.995495 (+ 191us) server_negotiation.cc:407] Connection header received
0927 16:28:51.995652 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:51.995655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:51.995707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:51.995806 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:51.996265 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.996791 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.997592 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:51.997825 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:51.997933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:52.000517 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:33351: BlockingRecv error: recv got EOF from 127.15.93.129:33351 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
Sep 27 16:28:52 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:52 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:52.193676 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:52.187928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:54529)
0927 16:28:52.188224 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:52.188240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:52.188398 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:52.188704 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:52.188708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:52.188930 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:52.189060 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:52.189068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.189910 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.189913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:52.190456 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:52.190463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.190871 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.190874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:52.190933 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:52.191588 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:52.191601 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:52.193560 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
W20260927 16:28:52.193840 24881 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:52.188095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:54529 (local address 127.15.93.190:44319)
0927 16:28:52.188229 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:52.188236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:52.188338 (+ 102us) server_negotiation.cc:407] Connection header received
0927 16:28:52.188502 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:52.188504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:52.188576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:52.188659 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:52.189200 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.189811 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.190570 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.190780 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.190852 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:52.193687 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:54529: BlockingRecv error: recv got EOF from 127.15.93.130:54529 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 16:28:52 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:52 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:52.377339 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:52.371815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:34785)
0927 16:28:52.372124 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:52.372137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:52.372268 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:52.372563 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:52.372566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:52.372769 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:52.372862 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:52.372867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.373625 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.373630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:52.374202 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:52.374209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.374692 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.374696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:52.374782 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:52.375254 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:52.375265 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:52.377232 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
W20260927 16:28:52.377521 24881 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:52.371890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:34785 (local address 127.15.93.190:44319)
0927 16:28:52.372072 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:52.372077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:52.372218 (+ 141us) server_negotiation.cc:407] Connection header received
0927 16:28:52.372368 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:52.372371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:52.372423 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:52.372543 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:52.372979 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.373514 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.374324 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:52.374578 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:52.374691 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:52.377378 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:34785: BlockingRecv error: recv got EOF from 127.15.93.131:34785 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:53.011016 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:53.001305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:53985)
0927 16:28:53.001702 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:53.001715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:53.001815 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:53.002054 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:53.002056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:53.002245 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:53.002336 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.002341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.003266 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.003270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.004110 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.004127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.004535 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.004538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.004595 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.008882 (+ 4287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:53.008895 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:53.010903 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":71,"threads_started":1}
W20260927 16:28:53.011217 24885 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:53.001349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:53985 (local address 127.15.93.190:44319)
0927 16:28:53.001724 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:53.001727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:53.001766 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:28:53.001896 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:53.001898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:53.001939 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:53.002015 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:53.002429 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.003109 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.004219 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.004442 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.004555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.011034 (+ 6479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:53985: BlockingRecv error: recv got EOF from 127.15.93.129:53985 (error 108)
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":218,"threads_started":1}
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:53.200093 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:53.194499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:42527)
0927 16:28:53.194784 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:53.194799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:53.194904 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:53.195146 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:53.195149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:53.195332 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:53.195466 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.195472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.196204 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.196207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.196831 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.196839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.197260 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.197262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.197332 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.197872 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:53.197885 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:53.199979 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
W20260927 16:28:53.200244 24885 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:53.194643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:42527 (local address 127.15.93.190:44319)
0927 16:28:53.194806 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:53.194815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:53.194871 (+ 56us) server_negotiation.cc:407] Connection header received
0927 16:28:53.194985 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:53.194988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:53.195031 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:53.195103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:53.195592 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.196093 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.196931 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.197151 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.197225 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.200117 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:42527: BlockingRecv error: recv got EOF from 127.15.93.130:42527 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:53 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:53.383574 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:53.378187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:40609)
0927 16:28:53.378454 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:53.378467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:53.378560 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:53.378775 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:53.378777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:53.378946 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:53.379040 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.379045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.379769 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.379771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.380318 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:53.380326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.380751 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.380754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:53.380823 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.381439 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:53.381460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:53.383362 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
W20260927 16:28:53.383608 24885 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:53.378250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:40609 (local address 127.15.93.190:44319)
0927 16:28:53.378361 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:53.378365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:53.378518 (+ 153us) server_negotiation.cc:407] Connection header received
0927 16:28:53.378625 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:53.378627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:53.378669 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:53.378737 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:53.379140 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.379668 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.380410 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:53.380625 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:53.380709 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:53.383504 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:40609: BlockingRecv error: recv got EOF from 127.15.93.131:40609 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:54.016855 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:54.011785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:58731)
0927 16:28:54.012041 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:54.012054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:54.012152 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:54.012381 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:54.012383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:54.012558 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:54.012649 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.012654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.013344 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.013346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.013894 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.013901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.014291 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.014293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.014354 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.014823 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:54.014834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:54.016747 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260927 16:28:54.017007 24889 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:54.011823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:58731 (local address 127.15.93.190:44319)
0927 16:28:54.012054 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:54.012057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:54.012121 (+ 64us) server_negotiation.cc:407] Connection header received
0927 16:28:54.012225 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:54.012227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:54.012277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:54.012352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:54.012739 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.013258 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.013983 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.014200 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.014271 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.016878 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:58731: BlockingRecv error: recv got EOF from 127.15.93.129:58731 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:54.206238 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:54.201017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:48199)
0927 16:28:54.201329 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:54.201343 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:54.201425 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:54.201665 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:54.201667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:54.201840 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:54.201931 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.201936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.202638 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.202640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.203191 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.203198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.203612 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.203616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.203716 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.204275 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:54.204289 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:54.206116 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
W20260927 16:28:54.206395 24889 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:54.201215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:48199 (local address 127.15.93.190:44319)
0927 16:28:54.201367 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:54.201371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:54.201387 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:28:54.201507 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:54.201510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:54.201552 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:54.201620 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:54.202026 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.202543 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.203286 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.203511 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.203584 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.206246 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:48199: BlockingRecv error: recv got EOF from 127.15.93.130:48199 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:54 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:54.390069 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:54.384486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:35987)
0927 16:28:54.384809 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:54.384822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:54.384958 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:54.385229 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:54.385231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:54.385435 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:54.385526 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.385534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.386289 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.386293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.386880 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:54.386889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.387421 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.387425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:54.387528 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.388103 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:54.388116 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:54.389957 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
W20260927 16:28:54.390208 24889 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:54.384547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:35987 (local address 127.15.93.190:44319)
0927 16:28:54.384704 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:54.384709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:54.384907 (+ 198us) server_negotiation.cc:407] Connection header received
0927 16:28:54.385057 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:54.385061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:54.385119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:54.385206 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:54.385647 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.386178 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.387004 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:54.387265 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:54.387418 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:54.390087 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:35987: BlockingRecv error: recv got EOF from 127.15.93.131:35987 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:55.023502 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:55.017661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:39021)
0927 16:28:55.017925 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:55.017938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:55.018052 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:55.018293 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:55.018295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:55.018479 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:55.018576 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.018582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.019283 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.019286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.020102 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.020113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.020782 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.020785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.020871 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.021440 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:55.021458 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:55.023315 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
W20260927 16:28:55.023717 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:55.017708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:39021 (local address 127.15.93.190:44319)
0927 16:28:55.017947 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:55.017951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:55.018020 (+ 69us) server_negotiation.cc:407] Connection header received
0927 16:28:55.018138 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:55.018140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:55.018184 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:55.018259 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:55.018672 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.019189 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.020213 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.020654 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.020767 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.023551 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:39021: BlockingRecv error: recv got EOF from 127.15.93.129:39021 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:55.212793 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:55.206957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:51345)
0927 16:28:55.207266 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:55.207278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:55.207430 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:55.207717 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:55.207720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:55.207961 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:55.208052 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.208058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.208912 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.208917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.209750 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.209760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.210340 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.210343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.210452 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.210914 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:55.210925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:55.212683 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
W20260927 16:28:55.212963 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:55.207024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:51345 (local address 127.15.93.190:44319)
0927 16:28:55.207192 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:55.207197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:55.207355 (+ 158us) server_negotiation.cc:407] Connection header received
0927 16:28:55.207552 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:55.207555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:55.207598 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:55.207697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:55.208181 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.208808 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.209869 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.210229 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.210323 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.212809 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:51345: BlockingRecv error: recv got EOF from 127.15.93.130:51345 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:55 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:55.396368 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:55.390826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:60855)
0927 16:28:55.391123 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:55.391136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:55.391232 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:55.391490 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:55.391492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:55.391670 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:55.391765 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.391770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.392626 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.392630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.393467 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:55.393476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.393893 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.393896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:55.393955 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.394442 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:55.394463 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:55.396257 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
W20260927 16:28:55.396498 24893 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:55.390924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:60855 (local address 127.15.93.190:44319)
0927 16:28:55.391064 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:55.391068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:55.391203 (+ 135us) server_negotiation.cc:407] Connection header received
0927 16:28:55.391312 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:55.391314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:55.391359 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:55.391450 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:55.391864 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.392511 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.393584 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:55.393802 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:55.393880 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:55.396381 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:60855: BlockingRecv error: recv got EOF from 127.15.93.131:60855 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:56.029824 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:56.024330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:44783)
0927 16:28:56.024718 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:56.024732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:56.024836 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:56.025197 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:56.025200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:56.025382 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:56.025479 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.025484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.026245 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.026248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.026801 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.026809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.027208 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.027211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.027269 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.027863 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:56.027875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:56.029709 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":89,"threads_started":1}
W20260927 16:28:56.030054 24897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:56.024385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:44783 (local address 127.15.93.190:44319)
0927 16:28:56.024639 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:56.024643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:56.024801 (+ 158us) server_negotiation.cc:407] Connection header received
0927 16:28:56.024972 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:56.024975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:56.025023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:56.025110 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:56.025621 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.026150 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.026900 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.027115 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.027187 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.029902 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:44783: BlockingRecv error: recv got EOF from 127.15.93.129:44783 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:56.219349 24898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:56.213561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:53317)
0927 16:28:56.213862 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:56.213875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:56.214011 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:56.214278 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:56.214281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:56.214499 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:56.214646 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.214655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.215465 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.215468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.216083 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.216092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.216552 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.216556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.216633 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.217181 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:56.217193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:56.219233 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":66,"threads_started":1}
W20260927 16:28:56.219655 24897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:56.213606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:53317 (local address 127.15.93.190:44319)
0927 16:28:56.213772 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:56.213777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:56.213949 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:28:56.214110 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:56.214113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:56.214169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:56.214257 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:56.214772 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.215303 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.216211 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.216438 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.216548 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.219366 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:53317: BlockingRecv error: recv got EOF from 127.15.93.130:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:56 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:56.403043 24899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:56.397124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:59423)
0927 16:28:56.397578 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:56.397592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:56.397742 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:56.398034 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:56.398037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:56.398258 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:56.398350 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.398356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.399178 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.399185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.399875 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:56.399884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.400379 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.400382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:56.400482 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.400975 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:56.400987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:56.402865 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":109,"threads_started":1}
W20260927 16:28:56.403115 24897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:56.397209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:59423 (local address 127.15.93.190:44319)
0927 16:28:56.397405 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:56.397412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:56.397683 (+ 271us) server_negotiation.cc:407] Connection header received
0927 16:28:56.397851 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:56.397857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:56.397913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:56.398017 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:56.398478 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.399058 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.400011 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:56.400257 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:56.400378 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:56.402999 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:59423: BlockingRecv error: recv got EOF from 127.15.93.131:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:57.035724 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:57.030509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:57473)
0927 16:28:57.030763 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:57.030775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:57.030885 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:57.031178 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:57.031181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:57.031358 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:57.031477 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.031483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.032230 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.032233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.032784 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.032791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.033172 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.033175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.033231 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.033692 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:57.033703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:57.035541 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":65,"threads_started":1}
W20260927 16:28:57.035801 24901 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:57.030739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:57473 (local address 127.15.93.190:44319)
0927 16:28:57.030971 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:57.030974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:57.030985 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:28:57.031019 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:57.031021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:57.031060 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:57.031138 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:57.031586 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.032131 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.032875 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.033084 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.033156 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.035676 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:57473: BlockingRecv error: recv got EOF from 127.15.93.129:57473 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:57.225664 24902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:57.220116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:45261)
0927 16:28:57.220417 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:57.220430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:57.220560 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:57.220878 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:57.220881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:57.221090 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:57.221195 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.221202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.222219 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.222222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.222771 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.222778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.223169 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.223172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.223234 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.223742 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:57.223753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:57.225537 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
W20260927 16:28:57.225827 24901 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:57.220197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:45261 (local address 127.15.93.190:44319)
0927 16:28:57.220342 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:57.220346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:57.220507 (+ 161us) server_negotiation.cc:407] Connection header received
0927 16:28:57.220679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:57.220682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:57.220738 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:57.220824 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:57.221287 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.222121 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.222868 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.223078 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.223154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.225675 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:45261: BlockingRecv error: recv got EOF from 127.15.93.130:45261 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:57 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:57.409298 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:57.403856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:39867)
0927 16:28:57.404164 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:57.404180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:57.404303 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:57.404553 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:57.404554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:57.404743 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:57.404839 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.404844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.405667 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.405669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.406231 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:57.406238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.406711 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.406713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:57.406774 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.407220 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:57.407230 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:57.409131 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
W20260927 16:28:57.409446 24901 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:57.403908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:39867 (local address 127.15.93.190:44319)
0927 16:28:57.404037 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:57.404043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:57.404263 (+ 220us) server_negotiation.cc:407] Connection header received
0927 16:28:57.404397 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:57.404400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:57.404455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:57.404534 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:57.404934 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.405558 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.406336 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:57.406606 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:57.406699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:57.409296 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:39867: BlockingRecv error: recv got EOF from 127.15.93.131:39867 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:58.045254 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:58.036612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:36927)
0927 16:28:58.036890 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:58.036909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:58.036994 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:58.037224 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:58.037226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:58.037395 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:58.037485 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.037490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.038310 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.038313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.038864 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.038872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.039276 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.039278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.039337 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.043128 (+ 3791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:58.043142 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:58.045120 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
W20260927 16:28:58.045383 24905 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:58.036706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:36927 (local address 127.15.93.190:44319)
0927 16:28:58.036969 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:58.036972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:58.036985 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:28:58.037072 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:58.037074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:58.037116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:58.037190 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:58.037573 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.038153 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.038963 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.039183 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.039257 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.045255 (+ 5998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:36927: BlockingRecv error: recv got EOF from 127.15.93.129:36927 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:58.231525 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:58.226372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:41537)
0927 16:28:58.226647 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:58.226660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:58.226755 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:58.226940 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:58.226942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:58.227111 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:58.227207 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.227212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.227931 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.227934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.228490 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.228498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.228887 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.228889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.228949 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.229401 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:58.229412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:58.231332 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
W20260927 16:28:58.231649 24905 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:58.226418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:41537 (local address 127.15.93.190:44319)
0927 16:28:58.226561 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:58.226566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:58.226728 (+ 162us) server_negotiation.cc:407] Connection header received
0927 16:28:58.226820 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:58.226822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:58.226866 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:58.226929 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:58.227298 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.227836 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.228581 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.228797 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.228870 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.231508 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:41537: BlockingRecv error: recv got EOF from 127.15.93.130:41537 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:58 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:58.415077 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:58.410036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:53285)
0927 16:28:58.410299 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:58.410312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:58.410397 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:58.410610 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:58.410612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:58.410792 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:58.410886 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.410891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.411596 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.411598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.412153 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:58.412162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.412550 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.412552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:58.412611 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.413076 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:58.413087 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:58.414908 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260927 16:28:58.415238 24905 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:58.410216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:53285 (local address 127.15.93.190:44319)
0927 16:28:58.410365 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:58.410371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:58.410418 (+ 47us) server_negotiation.cc:407] Connection header received
0927 16:28:58.410465 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:58.410467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:58.410510 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:58.410575 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:58.410983 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.411513 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.412252 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:58.412461 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:58.412533 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:58.415066 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:53285: BlockingRecv error: recv got EOF from 127.15.93.131:53285 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:59.051054 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:59.046031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:45321)
0927 16:28:59.046316 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:59.046329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:59.046411 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:59.046639 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:59.046641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:59.046814 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:59.046903 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.046908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.047624 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.047628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.048181 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.048189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.048579 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.048581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.048642 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.049090 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:59.049100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:59.050889 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
W20260927 16:28:59.051126 24909 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:59.046095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:45321 (local address 127.15.93.190:44319)
0927 16:28:59.046360 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:59.046364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:59.046376 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:28:59.046488 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:59.046490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:59.046531 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:59.046606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:59.046999 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.047536 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.048276 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.048490 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.048562 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.051005 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:45321: BlockingRecv error: recv got EOF from 127.15.93.129:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:59.237663 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:59.232278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:51527)
0927 16:28:59.232579 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:59.232600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:59.232736 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:59.233005 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:59.233008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:59.233220 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:59.233311 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.233316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.234090 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.234093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.234666 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.234674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.235150 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.235154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.235241 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.235761 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:59.235773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:59.237551 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
W20260927 16:28:59.237908 24909 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:59.232370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:51527 (local address 127.15.93.190:44319)
0927 16:28:59.232553 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:59.232558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:59.232676 (+ 118us) server_negotiation.cc:407] Connection header received
0927 16:28:59.232835 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:59.232840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:59.232892 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:59.232981 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:59.233429 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.233973 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.234803 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.235038 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.235145 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.237680 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:51527: BlockingRecv error: recv got EOF from 127.15.93.130:51527 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:28:59 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:28:59.421393 24911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:28:59.415885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:41991)
0927 16:28:59.416203 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 16:28:59.416219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:28:59.416321 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:28:59.416594 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:28:59.416597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:28:59.416806 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:28:59.416896 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.416902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.417663 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.417668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.418277 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:28:59.418285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.418762 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.418765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:28:59.418850 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.419436 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:28:59.419457 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:28:59.421280 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
W20260927 16:28:59.421528 24909 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:28:59.416055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:41991 (local address 127.15.93.190:44319)
0927 16:28:59.416203 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:28:59.416209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:28:59.416274 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:28:59.416427 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:28:59.416431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:28:59.416483 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:28:59.416569 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:28:59.417024 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.417553 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.418397 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:28:59.418644 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:28:59.418758 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:28:59.421408 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:41991: BlockingRecv error: recv got EOF from 127.15.93.131:41991 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:00.057778 24912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:00.051889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:40683)
0927 16:29:00.052276 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:00.052294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:00.052414 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:00.052734 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:00.052738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:00.052983 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:00.053078 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.053084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.053986 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.053990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.054563 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.054571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.055058 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.055062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.055148 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.055696 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:00.055713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:00.057668 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":89,"threads_started":1}
W20260927 16:29:00.057943 24913 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:00.051933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:40683 (local address 127.15.93.190:44319)
0927 16:29:00.052295 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:00.052298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:00.052351 (+ 53us) server_negotiation.cc:407] Connection header received
0927 16:29:00.052548 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:00.052551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:00.052611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:00.052712 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:00.053225 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.053875 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.054696 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.054936 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.055045 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.057805 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:40683: BlockingRecv error: recv got EOF from 127.15.93.129:40683 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:00.243844 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:00.238376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:52821)
0927 16:29:00.238665 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:00.238684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:00.238777 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:00.238989 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:00.238991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:00.239180 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:00.239301 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.239307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.240193 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.240197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.240743 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.240751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.241211 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.241215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.241284 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.241791 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:00.241803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:00.243727 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
W20260927 16:29:00.243983 24913 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:00.238416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:52821 (local address 127.15.93.190:44319)
0927 16:29:00.238556 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:00.238560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:00.238749 (+ 189us) server_negotiation.cc:407] Connection header received
0927 16:29:00.238842 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:00.238845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:00.238886 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:00.238955 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:00.239494 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.240055 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.240866 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.241082 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.241168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.243865 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:52821: BlockingRecv error: recv got EOF from 127.15.93.130:52821 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:00 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:00.427286 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:00.422137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:41533)
0927 16:29:00.422388 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:00.422399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:00.422492 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:00.422719 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:00.422721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:00.422892 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:00.422987 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.422992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.423740 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.423743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.424301 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:00.424309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.424796 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.424799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:00.424862 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.425424 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:00.425435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:00.427172 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260927 16:29:00.427476 24913 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:00.422230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:41533 (local address 127.15.93.190:44319)
0927 16:29:00.422359 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:00.422362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:00.422465 (+ 103us) server_negotiation.cc:407] Connection header received
0927 16:29:00.422571 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:00.422573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:00.422617 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:00.422684 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:00.423090 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.423637 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.424442 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:00.424691 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:00.424781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:00.427300 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:41533: BlockingRecv error: recv got EOF from 127.15.93.131:41533 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:01.063735 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:01.058494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:47807)
0927 16:29:01.058765 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:01.058783 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:01.058880 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:01.059133 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:01.059135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:01.059311 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:01.059467 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.059474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.060212 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.060214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.060752 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.060759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.061179 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.061181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.061240 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.061705 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:01.061716 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:01.063620 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
W20260927 16:29:01.063992 24917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:01.058606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47807 (local address 127.15.93.190:44319)
0927 16:29:01.058869 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:01.058872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:01.058885 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:29:01.058973 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:01.058975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:01.059016 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:01.059085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:01.059580 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.060120 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.060851 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.061084 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.061163 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.063804 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:47807: BlockingRecv error: recv got EOF from 127.15.93.129:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:01.250216 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:01.244750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:55723)
0927 16:29:01.245027 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:01.245041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:01.245147 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:01.245395 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:01.245399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:01.245601 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:01.245698 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.245703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.246400 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.246402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.246945 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.246952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.247380 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.247406 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.247490 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.248061 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:01.248078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:01.250039 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
W20260927 16:29:01.250272 24917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:01.244806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55723 (local address 127.15.93.190:44319)
0927 16:29:01.244953 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:01.244958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:01.245111 (+ 153us) server_negotiation.cc:407] Connection header received
0927 16:29:01.245227 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:01.245230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:01.245274 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:01.245372 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:01.245796 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.246312 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.247035 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.247245 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.247318 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.250165 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:55723: BlockingRecv error: recv got EOF from 127.15.93.130:55723 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:01 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:01.433516 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:01.427993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:60919)
0927 16:29:01.428242 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:01.428255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:01.428355 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:01.428576 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:01.428578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:01.428771 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:01.428865 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.428870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.429746 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.429751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.430471 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:01.430480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.430978 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.430981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:01.431044 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.431583 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:01.431597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:01.433396 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
W20260927 16:29:01.433647 24917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:01.428076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:60919 (local address 127.15.93.190:44319)
0927 16:29:01.428251 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:01.428254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:01.428321 (+ 67us) server_negotiation.cc:407] Connection header received
0927 16:29:01.428429 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:01.428431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:01.428477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:01.428543 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:01.428957 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.429634 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.430590 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:01.430882 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:01.430959 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:01.433531 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:60919: BlockingRecv error: recv got EOF from 127.15.93.131:60919 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:02.070003 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:02.064514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:44123)
0927 16:29:02.064785 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:02.064799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:02.064896 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:02.065132 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:02.065134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:02.065312 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:02.065406 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.065410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.066100 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.066102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.066673 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.066682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.067086 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.067088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.067148 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.067723 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:02.067734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:02.069824 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
W20260927 16:29:02.070060 24921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:02.064558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:44123 (local address 127.15.93.190:44319)
0927 16:29:02.064801 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:02.064804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:02.064869 (+ 65us) server_negotiation.cc:407] Connection header received
0927 16:29:02.064974 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:02.064976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:02.065018 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:02.065098 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:02.065499 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.066013 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.066776 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.066992 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.067068 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.069939 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:44123: BlockingRecv error: recv got EOF from 127.15.93.129:44123 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:02.257921 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:02.251004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:37513)
0927 16:29:02.251268 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:02.251285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:02.251416 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:02.251940 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:02.251945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:02.252150 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:02.252244 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.252249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.253568 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.253573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.254252 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.254261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.254991 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.254994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.255062 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.255706 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:02.255718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:02.257796 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
W20260927 16:29:02.258176 24921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:02.251218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37513 (local address 127.15.93.190:44319)
0927 16:29:02.251355 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:02.251360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:02.251395 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:29:02.251731 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:02.251736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:02.251813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:02.251918 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:02.252362 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.253370 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.254369 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.254860 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.254994 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.258002 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:37513: BlockingRecv error: recv got EOF from 127.15.93.130:37513 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:02 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:02.439723 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:02.434304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:36497)
0927 16:29:02.434693 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:02.434705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:02.434828 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:02.435155 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:02.435158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:02.435345 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:02.435487 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.435493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.436197 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.436199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.436752 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:02.436759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.437167 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.437170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:02.437238 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.437770 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:02.437797 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:02.439590 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":111,"threads_started":1}
W20260927 16:29:02.439831 24921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:02.434423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:36497 (local address 127.15.93.190:44319)
0927 16:29:02.434559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:02.434567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:02.434781 (+ 214us) server_negotiation.cc:407] Connection header received
0927 16:29:02.434935 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:02.434938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:02.435018 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:02.435103 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:02.435582 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.436096 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.436847 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:02.437057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:02.437139 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:02.439716 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:36497: BlockingRecv error: recv got EOF from 127.15.93.131:36497 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:03.079308 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:03.070917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:34319)
0927 16:29:03.071289 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:03.071302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:03.071417 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:03.071639 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:03.071641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:03.071815 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:03.071910 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.071915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.072661 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.072664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.073238 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.073246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.073676 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.073679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.073741 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.077356 (+ 3615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:03.077369 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:03.079178 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":99,"threads_started":1}
W20260927 16:29:03.079530 24925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:03.070976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:34319 (local address 127.15.93.190:44319)
0927 16:29:03.071301 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:03.071304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:03.071351 (+ 47us) server_negotiation.cc:407] Connection header received
0927 16:29:03.071482 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:03.071484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:03.071525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:03.071597 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:03.072003 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.072523 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.073346 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.073566 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.073659 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.079350 (+ 5691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:34319: BlockingRecv error: recv got EOF from 127.15.93.129:34319 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":60,"threads_started":1}
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:03.263581 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:03.258633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:33471)
0927 16:29:03.258929 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:03.258942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:03.259033 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:03.259256 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:03.259258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:03.259460 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:03.259555 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.259561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.260275 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.260277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.260840 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.260847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.261242 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.261244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.261301 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.261753 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:03.261764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:03.263458 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
W20260927 16:29:03.263667 24925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:03.258689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:33471 (local address 127.15.93.190:44319)
0927 16:29:03.258838 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:03.258842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:03.259004 (+ 162us) server_negotiation.cc:407] Connection header received
0927 16:29:03.259098 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:03.259100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:03.259154 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:03.259220 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:03.259667 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.260188 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.260932 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.261151 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.261223 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.263580 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:33471: BlockingRecv error: recv got EOF from 127.15.93.130:33471 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:03 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.131@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:03.446246 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:03.440504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.131:36105)
0927 16:29:03.440850 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:03.440864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:03.441010 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:03.441288 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:03.441291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:03.441544 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:03.441639 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.441646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.442426 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.442431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.443020 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:03.443028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.443573 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.443577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:03.443667 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.444236 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:03.444254 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:03.446126 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
W20260927 16:29:03.446419 24925 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:03.440638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:36105 (local address 127.15.93.190:44319)
0927 16:29:03.440801 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:03.440806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:03.440961 (+ 155us) server_negotiation.cc:407] Connection header received
0927 16:29:03.441116 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:03.441120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:03.441171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:03.441266 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:03.441763 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.442307 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.443142 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:03.443450 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:03.443570 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:03.446265 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.131:36105: BlockingRecv error: recv got EOF from 127.15.93.131:36105 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 16:29:04 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:04 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.129@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:04.085031 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.080171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.129:59755)
0927 16:29:04.080447 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:04.080459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:04.080541 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:04.080781 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:04.080783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:04.080957 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:04.081045 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.081050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.081752 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.081754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.082307 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.082314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.082711 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.082714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.082772 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.083229 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:04.083240 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:04.084914 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
W20260927 16:29:04.085175 24929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:04.080283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:59755 (local address 127.15.93.190:44319)
0927 16:29:04.080504 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:04.080508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:04.080520 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:29:04.080628 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:04.080631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:04.080673 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:04.080749 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:04.081136 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.081665 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.082396 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.082612 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.082694 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.085044 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.129:59755: BlockingRecv error: recv got EOF from 127.15.93.129:59755 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
Sep 27 16:29:04 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 16:29:04 dist-test-slave-grzf krb5kdc[23871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.93.130@KRBTEST.COM for oryx/127.15.93.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 16:29:04.269806 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.264679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44319 (local address 127.15.93.130:59969)
0927 16:29:04.264947 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:04.264960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:04.265047 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:04.265265 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:04.265267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:04.265436 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:04.265548 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.265554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.266291 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.266293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.266831 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.266838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.267231 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.267233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.267294 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.267783 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:04.267795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:04.269593 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.93.190:44319: Server oryx/127.15.93.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
W20260927 16:29:04.270026 24929 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:04.264758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:59969 (local address 127.15.93.190:44319)
0927 16:29:04.264912 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:04.264916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:04.265007 (+ 91us) server_negotiation.cc:407] Connection header received
0927 16:29:04.265121 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:04.265123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:04.265163 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:04.265225 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:04.265663 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.266201 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.266925 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.267140 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.267212 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.269802 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.93.130:59969: BlockingRecv error: recv got EOF from 127.15.93.130:59969 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:29:04.349255 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24454
I20260927 16:29:04.355173 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24606
I20260927 16:29:04.360443 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24738
I20260927 16:29:04.365770 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24384
2026-09-27T16:29:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17907 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T16:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:29:04Z Disabled control of system clock
I20260927 16:29:04.395095 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44995
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 16:29:04.500752 24934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:04.500991 24934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:04.501024 24934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:04.504868 24934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:29:04.504911 24934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:04.504926 24934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:04.504942 24934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:29:04.504957 24934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:29:04.509454 24934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:04.510375 24934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:04.511633 24934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:04.519753 24939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:04.519778 24940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:04.519867 24934 server_base.cc:1034] running on GCE node
W20260927 16:29:04.519791 24942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:04.520478 24934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:04.521497 24934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:04.522657 24934 hybrid_clock.cc:648] HybridClock initialized: now 1790526544522638 us; error 31 us; skew 500 ppm
I20260927 16:29:04.524472 24934 webserver.cc:460] Webserver started at http://127.15.93.190:42069/ using document root <none> and password file <none>
I20260927 16:29:04.525023 24934 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:04.525084 24934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:04.525324 24934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:04.527238 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "8c4f855c39764a6d8726267dc7112ff6"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "0755056738aeeed475347e72f78eaeb0"
server_key_iv: "51d34a46197d5eb89064d9909aa16d00"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.527848 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "8c4f855c39764a6d8726267dc7112ff6"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "0755056738aeeed475347e72f78eaeb0"
server_key_iv: "51d34a46197d5eb89064d9909aa16d00"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.531445 24934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 16:29:04.533660 24948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.534844 24934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:29:04.534976 24934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "8c4f855c39764a6d8726267dc7112ff6"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "0755056738aeeed475347e72f78eaeb0"
server_key_iv: "51d34a46197d5eb89064d9909aa16d00"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.535079 24934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:04.561491 24934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:04.562202 24934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:04.562393 24934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:04.568967 24934 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:44995
I20260927 16:29:04.568996 25000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:44995 every 8 connection(s)
I20260927 16:29:04.569870 24934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:29:04.570182 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 24934
I20260927 16:29:04.570297 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:29:04.570626 15734 external_mini_cluster.cc:1428] Setting key 2d7f2f4d1284c4fe5f1e5458dda4849a
I20260927 16:29:04.573256 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:04.579464 25001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6: Bootstrap starting.
I20260927 16:29:04.579749 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.572055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.15.93.190:44995)
0927 16:29:04.572667 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:04.572682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:04.572716 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:29:04.573369 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:04.573396 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:04.573790 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:04.574059 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:04.574375 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.575315 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.576583 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.577031 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.577146 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.578013 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:04.578017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:04.578029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:04.578387 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:04.578437 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:04.578454 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:04.578527 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:04.578712 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 16:29:04.578982 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
I20260927 16:29:04.581506 25001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:04.582283 25001 log.cc:824] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:04.584120 25001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6: No bootstrap required, opened a new log
I20260927 16:29:04.587477 25001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } }
I20260927 16:29:04.587671 25001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:04.587704 25001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4f855c39764a6d8726267dc7112ff6, State: Initialized, Role: FOLLOWER
I20260927 16:29:04.588063 25001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } }
I20260927 16:29:04.588181 25001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:29:04.588224 25001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:29:04.588291 25001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:04.589005 25001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } }
I20260927 16:29:04.589313 25001 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4f855c39764a6d8726267dc7112ff6; no voters:
I20260927 16:29:04.589529 25001 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:29:04.589646 25006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:04.589876 25006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4f855c39764a6d8726267dc7112ff6, State: Running, Role: LEADER
I20260927 16:29:04.590230 25006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } }
I20260927 16:29:04.590718 25001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:29:04.592047 25008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4f855c39764a6d8726267dc7112ff6. Latest consensus state: current_term: 1 leader_uuid: "8c4f855c39764a6d8726267dc7112ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } } }
I20260927 16:29:04.592166 25008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:04.592393 25007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4f855c39764a6d8726267dc7112ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4f855c39764a6d8726267dc7112ff6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44995 } } }
I20260927 16:29:04.592486 25007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:04.592545 25015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:29:04.595469 25015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:29:04.600384 25015 catalog_manager.cc:1347] Generated new cluster ID: bb903ef223fa4a6bae306a61dd6c7628
I20260927 16:29:04.600453 25015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:29:04.612640 25015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:29:04.613462 25015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:29:04.625118 25015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4f855c39764a6d8726267dc7112ff6: Generated new TSK 0
I20260927 16:29:04.625757 25015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:29:04.629948 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--builtin_ntp_servers=127.15.93.148:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 16:29:04.737360 25025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:04.737587 25025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:04.737659 25025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:04.741683 25025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:04.741730 25025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:04.741780 25025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:29:04.746307 25025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:04.747211 25025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:04.748504 25025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:04.758116 25033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:04.758100 25031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:04.758144 25030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:04.758473 25025 server_base.cc:1034] running on GCE node
I20260927 16:29:04.758901 25025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:04.759538 25025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:04.760716 25025 hybrid_clock.cc:648] HybridClock initialized: now 1790526544760707 us; error 23 us; skew 500 ppm
I20260927 16:29:04.762387 25025 webserver.cc:460] Webserver started at http://127.15.93.129:43739/ using document root <none> and password file <none>
I20260927 16:29:04.762948 25025 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:04.763038 25025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:04.763252 25025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:04.765268 25025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "16be06a1c0bd4c8ebffeedcac556b411"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "2b1e26b37429f5c5faf8e15331750574"
server_key_iv: "e3313aff9ed3fa3c5fc8a59ebf321f78"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.765828 25025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "16be06a1c0bd4c8ebffeedcac556b411"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "2b1e26b37429f5c5faf8e15331750574"
server_key_iv: "e3313aff9ed3fa3c5fc8a59ebf321f78"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.769274 25025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:29:04.771557 25039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.772703 25025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:29:04.772830 25025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "16be06a1c0bd4c8ebffeedcac556b411"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "2b1e26b37429f5c5faf8e15331750574"
server_key_iv: "e3313aff9ed3fa3c5fc8a59ebf321f78"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.772945 25025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:04.788463 25025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:04.789039 25025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:04.789224 25025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:04.789737 25025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:04.790577 25025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:04.790647 25025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.790714 25025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:04.790748 25025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.799418 25025 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:39013
I20260927 16:29:04.799458 25152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:39013 every 8 connection(s)
I20260927 16:29:04.800267 25025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:29:04.805989 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25025
I20260927 16:29:04.806114 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:29:04.806396 15734 external_mini_cluster.cc:1428] Setting key 01340c995e03dfefd0d2cb791b5f2f5e
I20260927 16:29:04.808363 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--builtin_ntp_servers=127.15.93.148:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:29:04.809509 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.801932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:36951 (local address 127.15.93.190:44995)
0927 16:29:04.802063 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:04.802067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:04.803070 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 16:29:04.804423 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:04.804426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:04.804475 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:04.804547 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:04.805363 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.806002 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.807469 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.807793 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.807896 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.808919 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:04.808926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:04.808932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:04.808949 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:04.808975 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:04.808988 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:04.809069 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:04.809204 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 16:29:04.809297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:29:04.810165 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.802315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44995 (local address 127.15.93.129:36951)
0927 16:29:04.802937 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:04.802975 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:04.804244 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:04.804716 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:04.804722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:04.804798 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:04.805247 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.805258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.806136 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.806140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.807310 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.807320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.807935 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.807937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.808085 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.808676 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:04.808678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:04.808714 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:04.808770 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:04.809072 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:04.809079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:04.809082 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:29:04.809170 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:29:04.809363 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
I20260927 16:29:04.811295 25153 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44995
I20260927 16:29:04.811533 25153 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:04.812088 25153 heartbeater.cc:511] Master 127.15.93.190:44995 requested a full tablet report, sending...
I20260927 16:29:04.813498 24965 ts_manager.cc:194] Registered new tserver with Master: 16be06a1c0bd4c8ebffeedcac556b411 (127.15.93.129:39013)
I20260927 16:29:04.814759 24965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.129:36951
W20260927 16:29:04.917498 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:04.917733 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:04.917794 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:04.922290 25156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:04.922343 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:04.922425 25156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:29:04.927059 25156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:04.928112 25156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:04.929415 25156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:04.938294 25162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:04.938411 25156 server_base.cc:1034] running on GCE node
W20260927 16:29:04.938539 25161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:04.938287 25164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:04.939061 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:04.939718 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:04.940898 25156 hybrid_clock.cc:648] HybridClock initialized: now 1790526544940891 us; error 28 us; skew 500 ppm
I20260927 16:29:04.942533 25156 webserver.cc:460] Webserver started at http://127.15.93.130:34181/ using document root <none> and password file <none>
I20260927 16:29:04.943078 25156 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:04.943172 25156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:04.943444 25156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:04.945351 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "c1439de7590c47b2b506924c60059b72"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "6f9f634f7455f3873e93c56f272be4cf"
server_key_iv: "341f46f17ecd10f88c921a83374340c9"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.945890 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "c1439de7590c47b2b506924c60059b72"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "6f9f634f7455f3873e93c56f272be4cf"
server_key_iv: "341f46f17ecd10f88c921a83374340c9"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.949391 25156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:29:04.951845 25170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.953001 25156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:29:04.953130 25156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "c1439de7590c47b2b506924c60059b72"
format_stamp: "Formatted at 2026-09-27 16:29:04 on dist-test-slave-grzf"
server_key: "6f9f634f7455f3873e93c56f272be4cf"
server_key_iv: "341f46f17ecd10f88c921a83374340c9"
server_key_version: "encryptionkey@0"
I20260927 16:29:04.953238 25156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:04.965230 25156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:04.965823 25156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:04.966007 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:04.966543 25156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:04.967464 25156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:04.967526 25156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.967589 25156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:04.967636 25156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:04.976176 25156 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:45427
I20260927 16:29:04.976264 25283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:45427 every 8 connection(s)
I20260927 16:29:04.977062 25156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:29:04.984973 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25156
I20260927 16:29:04.985090 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:29:04.985430 15734 external_mini_cluster.cc:1428] Setting key 45b549655e7fd9ad14b9ef450d01cee5
I20260927 16:29:04.986143 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.978741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:42105 (local address 127.15.93.190:44995)
0927 16:29:04.978877 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:04.978881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:04.979867 (+ 986us) server_negotiation.cc:407] Connection header received
0927 16:29:04.981231 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:04.981235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:04.981292 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:04.981400 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:04.982249 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.982780 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.983966 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.984185 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.984298 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.985522 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:04.985528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:04.985532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:04.985549 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:04.985572 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:04.985585 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:04.985650 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:04.985931 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 16:29:04.985984 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:29:04.987038 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:04.979112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44995 (local address 127.15.93.130:42105)
0927 16:29:04.979721 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:04.979758 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:04.981019 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:04.981564 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:04.981570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:04.981651 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:04.982091 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.982102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.982907 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.982911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.983833 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:04.983841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:04.984333 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:04.984337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:04.984487 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:04.985161 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:04.985165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:04.985240 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:04.985328 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:04.985726 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:04.985734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:04.985739 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:29:04.985856 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 16:29:04.986143 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260927 16:29:04.987493 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--builtin_ntp_servers=127.15.93.148:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 16:29:04.989791 25284 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44995
I20260927 16:29:04.990008 25284 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:04.990530 25284 heartbeater.cc:511] Master 127.15.93.190:44995 requested a full tablet report, sending...
I20260927 16:29:04.991554 24965 ts_manager.cc:194] Registered new tserver with Master: c1439de7590c47b2b506924c60059b72 (127.15.93.130:45427)
I20260927 16:29:04.992080 24965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.130:42105
W20260927 16:29:05.094743 25287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:05.095001 25287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:05.095062 25287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:05.099048 25287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:05.099134 25287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:05.099225 25287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:29:05.104027 25287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:44995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:05.105036 25287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:05.106348 25287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:05.115593 25293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.115763 25287 server_base.cc:1034] running on GCE node
W20260927 16:29:05.115607 25292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.115604 25295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.116273 25287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:05.116873 25287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:05.118049 25287 hybrid_clock.cc:648] HybridClock initialized: now 1790526545118035 us; error 26 us; skew 500 ppm
I20260927 16:29:05.119813 25287 webserver.cc:460] Webserver started at http://127.15.93.131:42035/ using document root <none> and password file <none>
I20260927 16:29:05.120366 25287 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:05.120453 25287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:05.120680 25287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:05.122625 25287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "98d196c740be4bc884aba024119547f1"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "090672e82f68a699b1e84d0b51522ef9"
server_key_iv: "1029d0d09fcde2cdbdc44c2b47bc8cd8"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.123204 25287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "98d196c740be4bc884aba024119547f1"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "090672e82f68a699b1e84d0b51522ef9"
server_key_iv: "1029d0d09fcde2cdbdc44c2b47bc8cd8"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.126996 25287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:29:05.129365 25301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.130528 25287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:29:05.130666 25287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "98d196c740be4bc884aba024119547f1"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "090672e82f68a699b1e84d0b51522ef9"
server_key_iv: "1029d0d09fcde2cdbdc44c2b47bc8cd8"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.130776 25287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:05.144714 25287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:05.145350 25287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:05.145529 25287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:05.146055 25287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:05.146940 25287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:05.147008 25287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.147074 25287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:05.147127 25287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.156436 25287 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:42029
I20260927 16:29:05.156458 25414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:42029 every 8 connection(s)
I20260927 16:29:05.157410 25287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:29:05.163738 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25287
I20260927 16:29:05.163825 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:29:05.164122 15734 external_mini_cluster.cc:1428] Setting key 232c58c205428cb39bc267217b7804d3
I20260927 16:29:05.166617 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.159180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:34397 (local address 127.15.93.190:44995)
0927 16:29:05.159311 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.159315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.160377 (+ 1062us) server_negotiation.cc:407] Connection header received
0927 16:29:05.161801 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.161804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.161860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.161960 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.162830 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:05.163356 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:05.164743 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:05.164971 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:05.165084 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:05.166221 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:05.166225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:05.166228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:05.166241 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:05.166263 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:05.166274 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:05.166337 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:05.166460 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 16:29:05.166512 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:29:05.167546 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.159613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:44995 (local address 127.15.93.131:34397)
0927 16:29:05.160234 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:05.160273 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:05.161587 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:05.162135 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:05.162142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:05.162224 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:05.162691 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:05.162703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:05.163554 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:05.163558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:05.164614 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:05.164624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:05.165116 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:05.165121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:05.165301 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:05.165907 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:05.165908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:05.165948 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:05.166054 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:05.166353 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:05.166359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:05.166364 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:29:05.166452 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:29:05.166721 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260927 16:29:05.168588 25415 heartbeater.cc:347] Connected to a master server at 127.15.93.190:44995
I20260927 16:29:05.168795 25415 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:05.169245 25415 heartbeater.cc:511] Master 127.15.93.190:44995 requested a full tablet report, sending...
I20260927 16:29:05.170301 24965 ts_manager.cc:194] Registered new tserver with Master: 98d196c740be4bc884aba024119547f1 (127.15.93.131:42029)
I20260927 16:29:05.170797 24965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.131:34397
I20260927 16:29:05.177772 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:29:05.180464 25004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:05.179538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.15.93.190:44995)
0927 16:29:05.179659 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.179663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.179855 (+ 192us) server_negotiation.cc:407] Connection header received
0927 16:29:05.179997 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.180001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.180042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.180135 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.180365 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57464: BlockingRecv error: recv got EOF from 127.0.0.1:57464 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:29:05.180769 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25025
I20260927 16:29:05.186288 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25156
I20260927 16:29:05.191586 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25287
I20260927 16:29:05.196633 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 24934
2026-09-27T16:29:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (829 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T16:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:29:05Z Disabled control of system clock
I20260927 16:29:05.224478 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:35343
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:35343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:29:05.329200 25426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:05.329434 25426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:05.329469 25426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:05.333425 25426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:29:05.333480 25426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:05.333495 25426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:05.333511 25426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:29:05.333573 25426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:29:05.338232 25426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:35343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:05.339346 25426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:05.340698 25426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:05.348783 25434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.348779 25431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.348832 25426 server_base.cc:1034] running on GCE node
W20260927 16:29:05.348774 25432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.349583 25426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:05.350699 25426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:05.351876 25426 hybrid_clock.cc:648] HybridClock initialized: now 1790526545351843 us; error 45 us; skew 500 ppm
I20260927 16:29:05.353571 25426 webserver.cc:460] Webserver started at http://127.15.93.190:41741/ using document root <none> and password file <none>
I20260927 16:29:05.354132 25426 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:05.354220 25426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:05.354458 25426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:05.356508 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "f162ea04f31b4a889f45c2b2ad57dcb8"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "164f747e700673eda29161d072092d7e"
server_key_iv: "a72a5aea451beee6a8f8e16d62e47c95"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.357055 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "f162ea04f31b4a889f45c2b2ad57dcb8"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "164f747e700673eda29161d072092d7e"
server_key_iv: "a72a5aea451beee6a8f8e16d62e47c95"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.360474 25426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:29:05.362644 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.363862 25426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:29:05.363997 25426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "f162ea04f31b4a889f45c2b2ad57dcb8"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "164f747e700673eda29161d072092d7e"
server_key_iv: "a72a5aea451beee6a8f8e16d62e47c95"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.364105 25426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:05.368422 25426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:05.368973 25426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:05.369150 25426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:05.375444 25426 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:35343
I20260927 16:29:05.375478 25492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:35343 every 8 connection(s)
I20260927 16:29:05.376262 25426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:29:05.379159 25493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:05.379256 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25426
I20260927 16:29:05.379391 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:29:05.379707 15734 external_mini_cluster.cc:1428] Setting key 3c655e545a2c59c788bb4bfa58230754
I20260927 16:29:05.384737 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.380989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.15.93.190:35343)
0927 16:29:05.381564 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.381578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.381610 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:29:05.382265 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.382281 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.382593 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.382907 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.383637 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:05.383642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:05.383653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:05.383663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:05.383708 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:05.383722 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:05.383865 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:05.383932 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 16:29:05.384152 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260927 16:29:05.385382 25493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8: Bootstrap starting.
I20260927 16:29:05.387709 25493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:05.388386 25493 log.cc:824] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:05.390003 25493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8: No bootstrap required, opened a new log
I20260927 16:29:05.393447 25493 raft_consensus.cc:348] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } }
I20260927 16:29:05.393639 25493 raft_consensus.cc:374] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:05.393672 25493 raft_consensus.cc:729] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f162ea04f31b4a889f45c2b2ad57dcb8, State: Initialized, Role: FOLLOWER
I20260927 16:29:05.394037 25493 consensus_queue.cc:260] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } }
I20260927 16:29:05.394234 25493 raft_consensus.cc:388] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:29:05.394310 25493 raft_consensus.cc:482] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:29:05.394408 25493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:05.395238 25493 raft_consensus.cc:504] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } }
I20260927 16:29:05.395565 25493 leader_election.cc:302] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f162ea04f31b4a889f45c2b2ad57dcb8; no voters:
I20260927 16:29:05.395792 25493 leader_election.cc:288] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:29:05.395917 25498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:05.396175 25498 raft_consensus.cc:686] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [term 1 LEADER]: Becoming Leader. State: Replica: f162ea04f31b4a889f45c2b2ad57dcb8, State: Running, Role: LEADER
I20260927 16:29:05.396502 25498 consensus_queue.cc:237] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } }
I20260927 16:29:05.396917 25493 sys_catalog.cc:564] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:29:05.398047 25500 sys_catalog.cc:455] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f162ea04f31b4a889f45c2b2ad57dcb8. Latest consensus state: current_term: 1 leader_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } } }
I20260927 16:29:05.398183 25500 sys_catalog.cc:458] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:05.398509 25499 sys_catalog.cc:455] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f162ea04f31b4a889f45c2b2ad57dcb8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 35343 } } }
I20260927 16:29:05.398583 25507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:29:05.398655 25499 sys_catalog.cc:458] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:05.401265 25507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:29:05.405985 25507 catalog_manager.cc:1347] Generated new cluster ID: 0f62688384c246cca2fa6542dbb8e916
I20260927 16:29:05.406044 25507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:29:05.414122 25507 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:29:05.414424 25507 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:29:05.422170 25507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f162ea04f31b4a889f45c2b2ad57dcb8: Generated new TSK 0
I20260927 16:29:05.422851 25507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:29:05.436826 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--builtin_ntp_servers=127.15.93.148:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:29:05.543442 25517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:05.543709 25517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:05.543778 25517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:05.547729 25517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:05.547793 25517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:05.547879 25517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:29:05.552582 25517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:05.553544 25517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:05.554874 25517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:05.563879 25523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.563875 25525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.564085 25517 server_base.cc:1034] running on GCE node
W20260927 16:29:05.563894 25522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.564575 25517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:05.565160 25517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:05.566339 25517 hybrid_clock.cc:648] HybridClock initialized: now 1790526545566324 us; error 26 us; skew 500 ppm
I20260927 16:29:05.568173 25517 webserver.cc:460] Webserver started at http://127.15.93.129:38577/ using document root <none> and password file <none>
I20260927 16:29:05.568732 25517 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:05.568819 25517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:05.569012 25517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:05.570919 25517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "98dd5700bda24279ba4cf503bda83144"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "c4e75cb6764c1d42bf4ef9cf181429b3"
server_key_iv: "23312cf25127f455811a7c37d36b738d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.571522 25517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "98dd5700bda24279ba4cf503bda83144"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "c4e75cb6764c1d42bf4ef9cf181429b3"
server_key_iv: "23312cf25127f455811a7c37d36b738d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.575012 25517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:29:05.577397 25531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.578521 25517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:29:05.578646 25517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "98dd5700bda24279ba4cf503bda83144"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "c4e75cb6764c1d42bf4ef9cf181429b3"
server_key_iv: "23312cf25127f455811a7c37d36b738d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.578750 25517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:05.583055 25517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:05.583664 25517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:05.583833 25517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:05.584321 25517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:05.585136 25517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:05.585203 25517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.585270 25517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:05.585312 25517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.593446 25517 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:38323
I20260927 16:29:05.593474 25644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:38323 every 8 connection(s)
I20260927 16:29:05.594240 25517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:29:05.600080 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.595966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:57637 (local address 127.15.93.190:35343)
0927 16:29:05.596092 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.596095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.597086 (+ 991us) server_negotiation.cc:407] Connection header received
0927 16:29:05.598535 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.598538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.598576 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.598652 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.599711 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:05.599714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:05.599716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:05.599727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:05.599746 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:05.599754 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:05.599830 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:05.599939 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 16:29:05.599976 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:29:05.600859 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.596330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35343 (local address 127.15.93.129:57637)
0927 16:29:05.596946 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:05.596983 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:05.598276 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:05.598800 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:05.598806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:05.598888 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:05.599466 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:05.599468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:05.599504 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:05.599572 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:05.599827 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:05.599833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:05.599837 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:29:05.599956 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 16:29:05.600154 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260927 16:29:05.602023 25645 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35343
I20260927 16:29:05.602216 25645 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:05.602545 25645 heartbeater.cc:511] Master 127.15.93.190:35343 requested a full tablet report, sending...
I20260927 16:29:05.602543 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25517
I20260927 16:29:05.602670 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:29:05.602958 15734 external_mini_cluster.cc:1428] Setting key eecd769c5c6637689564d3e5323e0399
I20260927 16:29:05.604095 25457 ts_manager.cc:194] Registered new tserver with Master: 98dd5700bda24279ba4cf503bda83144 (127.15.93.129:38323)
I20260927 16:29:05.604961 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--builtin_ntp_servers=127.15.93.148:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 16:29:05.726424 25648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:05.726812 25648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:05.726905 25648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:05.733374 25648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:05.733423 25648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:05.733479 25648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:29:05.738095 25648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:05.739033 25648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:05.740468 25648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:05.749333 25654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.749454 25648 server_base.cc:1034] running on GCE node
W20260927 16:29:05.749473 25653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.749662 25656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.750044 25648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:05.750634 25648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:05.751788 25648 hybrid_clock.cc:648] HybridClock initialized: now 1790526545751775 us; error 26 us; skew 500 ppm
I20260927 16:29:05.753419 25648 webserver.cc:460] Webserver started at http://127.15.93.130:39887/ using document root <none> and password file <none>
I20260927 16:29:05.753954 25648 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:05.754012 25648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:05.754208 25648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:05.756246 25648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "65dda57a9feb4c0eab1f264a88e98be2"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "a02a14553552ee75c1889bc05f8ebc47"
server_key_iv: "d9842d29ddc1179ebecb6177afbfcb2d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.756803 25648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "65dda57a9feb4c0eab1f264a88e98be2"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "a02a14553552ee75c1889bc05f8ebc47"
server_key_iv: "d9842d29ddc1179ebecb6177afbfcb2d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.760309 25648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:29:05.762717 25662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.763969 25648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:29:05.764096 25648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "65dda57a9feb4c0eab1f264a88e98be2"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "a02a14553552ee75c1889bc05f8ebc47"
server_key_iv: "d9842d29ddc1179ebecb6177afbfcb2d"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.764204 25648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:05.768673 25648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:05.769246 25648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:05.769426 25648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:05.769908 25648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:05.770718 25648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:05.770785 25648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.770843 25648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:05.770891 25648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.779315 25648 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34799
I20260927 16:29:05.779340 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34799 every 8 connection(s)
I20260927 16:29:05.780115 25648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:29:05.780371 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25648
I20260927 16:29:05.780514 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:29:05.780836 15734 external_mini_cluster.cc:1428] Setting key 8a003e7f1f78c45feba2b1ea75a4966d
I20260927 16:29:05.782941 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--builtin_ntp_servers=127.15.93.148:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 16:29:05.786749 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.781968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:50101 (local address 127.15.93.190:35343)
0927 16:29:05.782104 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.782108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.783305 (+ 1197us) server_negotiation.cc:407] Connection header received
0927 16:29:05.784916 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.784919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.784954 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.785026 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.786222 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:05.786228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:05.786234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:05.786251 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:05.786278 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:05.786291 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:05.786387 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:05.786507 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 16:29:05.786555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:29:05.787503 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.782483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35343 (local address 127.15.93.130:50101)
0927 16:29:05.783177 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:05.783214 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:05.784688 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:05.785199 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:05.785210 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:05.785333 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:05.785975 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:05.785977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:05.786015 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:05.786079 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:05.786380 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:05.786386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:05.786390 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:29:05.786512 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 16:29:05.786700 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260927 16:29:05.788532 25776 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35343
I20260927 16:29:05.788791 25776 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:05.789273 25776 heartbeater.cc:511] Master 127.15.93.190:35343 requested a full tablet report, sending...
I20260927 16:29:05.790398 25457 ts_manager.cc:194] Registered new tserver with Master: 65dda57a9feb4c0eab1f264a88e98be2 (127.15.93.130:34799)
W20260927 16:29:05.890515 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:05.890746 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:05.890784 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:05.894701 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:05.894750 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:05.894802 25779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:29:05.899734 25779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:05.900671 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:05.901933 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:05.911988 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.911996 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:05.912000 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:05.912263 25779 server_base.cc:1034] running on GCE node
I20260927 16:29:05.912662 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:05.913199 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:05.914356 25779 hybrid_clock.cc:648] HybridClock initialized: now 1790526545914342 us; error 27 us; skew 500 ppm
I20260927 16:29:05.916146 25779 webserver.cc:460] Webserver started at http://127.15.93.131:46317/ using document root <none> and password file <none>
I20260927 16:29:05.916695 25779 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:05.916755 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:05.916970 25779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:05.918855 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "a52b7b65f9624fb9b41465c026c43b66"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "edccc57e58208f13ad046041ffcd44c5"
server_key_iv: "f7691f2685e32a5644cade4986161fb9"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.919339 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "a52b7b65f9624fb9b41465c026c43b66"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "edccc57e58208f13ad046041ffcd44c5"
server_key_iv: "f7691f2685e32a5644cade4986161fb9"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.922708 25779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:29:05.924994 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.926105 25779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:29:05.926225 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "a52b7b65f9624fb9b41465c026c43b66"
format_stamp: "Formatted at 2026-09-27 16:29:05 on dist-test-slave-grzf"
server_key: "edccc57e58208f13ad046041ffcd44c5"
server_key_iv: "f7691f2685e32a5644cade4986161fb9"
server_key_version: "encryptionkey@0"
I20260927 16:29:05.926326 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:05.930581 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:05.931131 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:05.931301 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:05.931821 25779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:05.932643 25779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:05.932694 25779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.932763 25779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:05.932801 25779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:05.941107 25779 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40477
I20260927 16:29:05.941143 25906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40477 every 8 connection(s)
I20260927 16:29:05.941890 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:29:05.947329 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.943510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:39003 (local address 127.15.93.190:35343)
0927 16:29:05.943651 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.943655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.944551 (+ 896us) server_negotiation.cc:407] Connection header received
0927 16:29:05.945863 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.945866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.945909 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.946004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.947011 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:05.947013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:05.947014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:29:05.947022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:05.947037 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:29:05.947044 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:05.947115 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:05.947201 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 16:29:05.947234 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:29:05.948129 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:05.943835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:35343 (local address 127.15.93.131:39003)
0927 16:29:05.944414 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:05.944450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:05.945629 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:05.946145 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:05.946151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:05.946228 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:05.946795 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:29:05.946796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:05.946831 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:29:05.946893 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:05.947094 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:05.947098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:05.947101 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 16:29:05.947215 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:29:05.947430 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260927 16:29:05.949208 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25779
I20260927 16:29:05.949236 25907 heartbeater.cc:347] Connected to a master server at 127.15.93.190:35343
I20260927 16:29:05.949337 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:29:05.949470 25907 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:05.949687 15734 external_mini_cluster.cc:1428] Setting key c7e6ef54720aa539872e4a6bd5e76eef
I20260927 16:29:05.949867 25907 heartbeater.cc:511] Master 127.15.93.190:35343 requested a full tablet report, sending...
I20260927 16:29:05.950863 25457 ts_manager.cc:194] Registered new tserver with Master: a52b7b65f9624fb9b41465c026c43b66 (127.15.93.131:40477)
I20260927 16:29:05.951752 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 16:29:05.954304 25496 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:29:05.953488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.15.93.190:35343)
0927 16:29:05.953642 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:05.953649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:05.953819 (+ 170us) server_negotiation.cc:407] Connection header received
0927 16:29:05.953901 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:05.953903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:05.953928 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:05.953991 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:05.954162 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52012: BlockingRecv error: recv got EOF from 127.0.0.1:52012 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:29:05.954882 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25517
I20260927 16:29:05.960592 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25648
I20260927 16:29:05.965574 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25779
I20260927 16:29:05.970662 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25426
2026-09-27T16:29:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (774 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:29:06 dist-test-slave-grzf krb5kdc[25917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:29:06 dist-test-slave-grzf krb5kdc[25917](info): set up 2 sockets
Sep 27 16:29:06 dist-test-slave-grzf krb5kdc[25917](info): commencing operation
krb5kdc: starting...
W20260927 16:29:07.008495 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:29:07.124257 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34961
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:07.229504 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:07.229847 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:07.229920 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:07.234058 25930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:29:07.234157 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:07.234198 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:07.234256 25930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:29:07.234297 25930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:29:07.239508 25930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:07.240470 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:07.241432 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:07.249477 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.249557 25930 server_base.cc:1034] running on GCE node
W20260927 16:29:07.249475 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.249460 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.250260 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:07.251235 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:07.252420 25930 hybrid_clock.cc:648] HybridClock initialized: now 1790526547252406 us; error 32 us; skew 500 ppm
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:07.254985 25930 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:29:07.255826 25930 webserver.cc:460] Webserver started at http://127.15.93.190:41607/ using document root <none> and password file <none>
I20260927 16:29:07.256301 25930 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:07.256353 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:07.256503 25930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:07.258352 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "e1fa6eb0d7da44ca964170955774cc10"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "193b75793b9216dfdf5f45c672474f67"
server_key_iv: "0adaa545f55bfff5fc1c72e2400aff24"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.258841 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "e1fa6eb0d7da44ca964170955774cc10"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "193b75793b9216dfdf5f45c672474f67"
server_key_iv: "0adaa545f55bfff5fc1c72e2400aff24"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.262220 25930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:29:07.264432 25945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.265537 25930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:29:07.265661 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "e1fa6eb0d7da44ca964170955774cc10"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "193b75793b9216dfdf5f45c672474f67"
server_key_iv: "0adaa545f55bfff5fc1c72e2400aff24"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.265753 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:07.277858 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:07.280432 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:07.280614 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:07.286970 25930 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:34961
I20260927 16:29:07.287240 25997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:34961 every 8 connection(s)
I20260927 16:29:07.287884 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:29:07.289359 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 25930
I20260927 16:29:07.289481 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:29:07.289785 15734 external_mini_cluster.cc:1428] Setting key 33115f5311b83cf5f5756fec586d654d
I20260927 16:29:07.290848 25998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:07.297122 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: Bootstrap starting.
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:07.299573 25998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:07.300386 25998 log.cc:824] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:07.302157 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: No bootstrap required, opened a new log
I20260927 16:29:07.303479 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.291072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.15.93.190:34961)
0927 16:29:07.291471 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:07.291486 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:07.291519 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:29:07.292160 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:07.292174 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:07.292464 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:07.292726 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:07.293597 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.294556 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.295706 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.296145 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.296255 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.298892 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:07.298896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:07.298908 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:07.298934 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:07.300711 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.301167 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.301172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.301176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.301280 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.301562 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.301566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.301568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.301940 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:07.302136 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:07.302477 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 16:29:07.302739 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260927 16:29:07.306140 25998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } }
I20260927 16:29:07.306368 25998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:07.306445 25998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fa6eb0d7da44ca964170955774cc10, State: Initialized, Role: FOLLOWER
I20260927 16:29:07.306880 25998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } }
I20260927 16:29:07.307029 25998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:29:07.307109 25998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:29:07.307231 25998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:07.308287 25998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } }
I20260927 16:29:07.308645 25998 leader_election.cc:302] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1fa6eb0d7da44ca964170955774cc10; no voters:
I20260927 16:29:07.308924 25998 leader_election.cc:288] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:29:07.309010 26003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:07.309288 26003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [term 1 LEADER]: Becoming Leader. State: Replica: e1fa6eb0d7da44ca964170955774cc10, State: Running, Role: LEADER
I20260927 16:29:07.309644 26003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } }
I20260927 16:29:07.310092 25998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:29:07.311224 26004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1fa6eb0d7da44ca964170955774cc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } } }
I20260927 16:29:07.311352 26004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:07.311672 26005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fa6eb0d7da44ca964170955774cc10. Latest consensus state: current_term: 1 leader_uuid: "e1fa6eb0d7da44ca964170955774cc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fa6eb0d7da44ca964170955774cc10" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34961 } } }
I20260927 16:29:07.311789 26005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10 [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:07.311731 26012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:29:07.314890 26012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:29:07.320165 26012 catalog_manager.cc:1347] Generated new cluster ID: e95e3c1e861940c5aa8407a5d774533b
I20260927 16:29:07.320225 26012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:29:07.331735 26012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:29:07.332557 26012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:29:07.338114 26012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: Generated new TSK 0
I20260927 16:29:07.338733 26012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:29:07.369823 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:07.487809 26024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:07.488042 26024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:07.488122 26024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:07.492192 26024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:07.492240 26024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:07.492327 26024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:29:07.497452 26024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:07.498425 26024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:07.499415 26024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:07.508220 26030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.508234 26029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.508229 26032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.508441 26024 server_base.cc:1034] running on GCE node
I20260927 16:29:07.508913 26024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:07.509483 26024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:07.510661 26024 hybrid_clock.cc:648] HybridClock initialized: now 1790526547510648 us; error 30 us; skew 500 ppm
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:07.513288 26024 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:29:07.514012 26024 webserver.cc:460] Webserver started at http://127.15.93.129:37985/ using document root <none> and password file <none>
I20260927 16:29:07.514501 26024 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:07.514555 26024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:07.514714 26024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:07.516566 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "6edccadedb14443b8facfcce9bbb53e4"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "77ec76d4fe8c1c7181a285d2d32659c0"
server_key_iv: "eee1d06c01a49a1483f3169da0ff4fed"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.517053 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "6edccadedb14443b8facfcce9bbb53e4"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "77ec76d4fe8c1c7181a285d2d32659c0"
server_key_iv: "eee1d06c01a49a1483f3169da0ff4fed"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.520387 26024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:29:07.522656 26039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.523875 26024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:29:07.523988 26024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "6edccadedb14443b8facfcce9bbb53e4"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "77ec76d4fe8c1c7181a285d2d32659c0"
server_key_iv: "eee1d06c01a49a1483f3169da0ff4fed"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.524087 26024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:07.547694 26024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:07.550314 26024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:07.550506 26024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:07.551062 26024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:07.551855 26024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:07.551899 26024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.551963 26024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:07.551994 26024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.560951 26024 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:43377
I20260927 16:29:07.560981 26152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:43377 every 8 connection(s)
I20260927 16:29:07.561789 26024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:29:07.566066 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 26024
I20260927 16:29:07.566213 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:29:07.566519 15734 external_mini_cluster.cc:1428] Setting key 5dc65cfed4a6365bab88aff8f90c73ea
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:07.576181 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.563480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:49527 (local address 127.15.93.190:34961)
0927 16:29:07.563615 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:07.563620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:07.564411 (+ 791us) server_negotiation.cc:407] Connection header received
0927 16:29:07.565801 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:07.565805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:07.565866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:07.565948 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:07.567331 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.568024 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.569286 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.569594 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.569698 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.572369 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:07.572374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:07.572378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:07.572412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:07.574086 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.574564 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.574569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.574573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.574675 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.574990 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.574993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.574995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.575232 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:07.575342 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:07.575858 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 16:29:07.576001 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:29:07.576893 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.563835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34961 (local address 127.15.93.129:49527)
0927 16:29:07.564264 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:07.564305 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:07.565587 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:07.566119 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:07.566126 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:07.566576 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:07.567161 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.567196 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.568149 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.568153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.569163 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.569173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.569764 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.569770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.569995 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.570682 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:07.570698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:07.572206 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:07.574193 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.574197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.574208 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.574464 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.574771 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.574773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.574774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.574896 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.575339 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:07.575343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:07.575609 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 16:29:07.575766 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 16:29:07.576005 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260927 16:29:07.578132 26153 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34961
I20260927 16:29:07.578405 26153 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:07.578908 26153 heartbeater.cc:511] Master 127.15.93.190:34961 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:29:07.580605 25962 ts_manager.cc:194] Registered new tserver with Master: 6edccadedb14443b8facfcce9bbb53e4 (127.15.93.129:43377)
I20260927 16:29:07.582031 25962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:49527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:29:07.593668 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:07.699903 26158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:07.700125 26158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:07.700165 26158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:07.704134 26158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:07.704185 26158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:07.704283 26158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:29:07.709350 26158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:07.710278 26158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:07.711299 26158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:07.720446 26164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.720485 26163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.720455 26166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.720530 26158 server_base.cc:1034] running on GCE node
I20260927 16:29:07.721004 26158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:07.721583 26158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:07.722764 26158 hybrid_clock.cc:648] HybridClock initialized: now 1790526547722751 us; error 24 us; skew 500 ppm
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:07.725353 26158 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:29:07.726130 26158 webserver.cc:460] Webserver started at http://127.15.93.130:39333/ using document root <none> and password file <none>
I20260927 16:29:07.726689 26158 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:07.726783 26158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:07.726991 26158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:07.729140 26158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "e415e7d2db464d3da9ac141d1a944345"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "f1e4342b6a86e7561e99774acc94c274"
server_key_iv: "edfdf31f77e3352467088cdf193c47f6"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.729699 26158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "e415e7d2db464d3da9ac141d1a944345"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "f1e4342b6a86e7561e99774acc94c274"
server_key_iv: "edfdf31f77e3352467088cdf193c47f6"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.733179 26158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:29:07.735452 26173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.736567 26158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:29:07.736693 26158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "e415e7d2db464d3da9ac141d1a944345"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "f1e4342b6a86e7561e99774acc94c274"
server_key_iv: "edfdf31f77e3352467088cdf193c47f6"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.736793 26158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:07.748311 26158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:07.750828 26158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:07.751019 26158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:07.751577 26158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:07.752321 26158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:07.752388 26158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.752454 26158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:07.752506 26158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.760818 26158 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:35967
I20260927 16:29:07.760851 26286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:35967 every 8 connection(s)
I20260927 16:29:07.761646 26158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:29:07.769060 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 26158
I20260927 16:29:07.769182 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:29:07.769526 15734 external_mini_cluster.cc:1428] Setting key dbce1e0140accd7c34b35d60e6bee85e
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:07.775221 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.763239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:46095 (local address 127.15.93.190:34961)
0927 16:29:07.763489 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:07.763493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:07.764169 (+ 676us) server_negotiation.cc:407] Connection header received
0927 16:29:07.765538 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:07.765541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:07.765597 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:07.765705 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:07.766789 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.767315 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.768503 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.768727 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.768859 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.771611 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:07.771617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:07.771623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:07.771656 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:07.773114 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.773709 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.773712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.773713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.773783 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.774143 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.774145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.774146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.774298 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:07.774369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:07.774954 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 16:29:07.775047 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 16:29:07.776311 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.763596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34961 (local address 127.15.93.130:46095)
0927 16:29:07.764033 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:07.764073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:07.765330 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:07.765868 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:07.765874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:07.766215 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:07.766649 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.766661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.767503 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.767508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.768373 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.768383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.768871 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.768875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.769081 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.769739 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:07.769754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:07.771457 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:07.773223 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.773230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.773246 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.773604 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.773869 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.773874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.773879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.774051 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.774400 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:07.774409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:07.774717 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 16:29:07.774919 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 16:29:07.775262 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260927 16:29:07.777550 26287 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34961
I20260927 16:29:07.777805 26287 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:07.778249 26287 heartbeater.cc:511] Master 127.15.93.190:34961 requested a full tablet report, sending...
I20260927 16:29:07.779222 25962 ts_manager.cc:194] Registered new tserver with Master: e415e7d2db464d3da9ac141d1a944345 (127.15.93.130:35967)
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:29:07.779840 25962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:46095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:29:07.794085 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:07.901002 26292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:07.901297 26292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:07.901371 26292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:07.905427 26292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:07.905512 26292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:07.905603 26292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:29:07.910778 26292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:07.912127 26292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:07.913406 26292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:07.922662 26298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:07.922655 26297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.922806 26292 server_base.cc:1034] running on GCE node
W20260927 16:29:07.922973 26300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:07.923348 26292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:07.923957 26292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:07.925109 26292 hybrid_clock.cc:648] HybridClock initialized: now 1790526547925080 us; error 42 us; skew 500 ppm
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:07.927748 26292 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:29:07.928658 26292 webserver.cc:460] Webserver started at http://127.15.93.131:36111/ using document root <none> and password file <none>
I20260927 16:29:07.929204 26292 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:07.929263 26292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:07.929484 26292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:29:07.931360 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "0e4fe5450e804dd6a3cc43b65856bffb"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "a6e0ad256b01abdc4a2a85dbcbd219c0"
server_key_iv: "fd02b77a21734d1599cbba4745c7365d"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.931958 26292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "0e4fe5450e804dd6a3cc43b65856bffb"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "a6e0ad256b01abdc4a2a85dbcbd219c0"
server_key_iv: "fd02b77a21734d1599cbba4745c7365d"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.935720 26292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:29:07.937861 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.939072 26292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:29:07.939162 26292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "0e4fe5450e804dd6a3cc43b65856bffb"
format_stamp: "Formatted at 2026-09-27 16:29:07 on dist-test-slave-grzf"
server_key: "a6e0ad256b01abdc4a2a85dbcbd219c0"
server_key_iv: "fd02b77a21734d1599cbba4745c7365d"
server_key_version: "encryptionkey@0"
I20260927 16:29:07.939235 26292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:07.949925 26292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:07.952536 26292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:07.952678 26292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:07.953161 26292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:07.953876 26292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:07.953924 26292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.953961 26292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:07.953979 26292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:07.962402 26292 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:46205
I20260927 16:29:07.962427 26420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:46205 every 8 connection(s)
I20260927 16:29:07.963253 26292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:29:07.969411 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 26292
I20260927 16:29:07.969543 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:29:07.969825 15734 external_mini_cluster.cc:1428] Setting key 8cca870f412b81f66000aff1e1f833ea
Sep 27 16:29:07 dist-test-slave-grzf krb5kdc[25917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526547, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:07.976245 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.964968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:53365 (local address 127.15.93.190:34961)
0927 16:29:07.965104 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:07.965108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:07.965900 (+ 792us) server_negotiation.cc:407] Connection header received
0927 16:29:07.967180 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:07.967183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:07.967228 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:07.967302 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:07.968391 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.968966 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.970298 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.970523 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.970656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.973032 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:07.973036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:07.973039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:07.973072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:07.974266 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.974762 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.974767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.974768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.974849 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.975173 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.975178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.975180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.975351 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:07.975484 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:07.975958 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 16:29:07.976071 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:29:07.976939 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.965327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34961 (local address 127.15.93.131:53365)
0927 16:29:07.965753 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:07.965793 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:07.966995 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:07.967473 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:07.967479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:07.967830 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:07.968271 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.968283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.969070 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.969073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.970174 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:07.970183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.970689 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.970693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:07.970868 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.971483 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:07.971499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:07.972875 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:07.974379 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.974385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.974398 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.974659 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.974946 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:07.974949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:07.974951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:07.975078 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:07.975503 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:07.975509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:07.975731 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 16:29:07.975904 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 16:29:07.976196 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
I20260927 16:29:07.977916 26421 heartbeater.cc:347] Connected to a master server at 127.15.93.190:34961
I20260927 16:29:07.978106 26421 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:07.978502 26421 heartbeater.cc:511] Master 127.15.93.190:34961 requested a full tablet report, sending...
I20260927 16:29:07.979444 25962 ts_manager.cc:194] Registered new tserver with Master: 0e4fe5450e804dd6a3cc43b65856bffb (127.15.93.131:46205)
I20260927 16:29:07.979978 25962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:53365
I20260927 16:29:07.983309 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:29:07.991632 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:07.985105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.15.93.190:34961)
0927 16:29:07.985233 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:07.985237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:07.985422 (+ 185us) server_negotiation.cc:407] Connection header received
0927 16:29:07.985510 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:07.985512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:07.985546 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:07.985603 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:07.986205 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.986681 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.987472 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:07.987682 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:07.987768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:07.988932 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:07.988935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:07.988937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:07.988958 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:07.990131 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.990539 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.990541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.990542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.990608 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:07.990844 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:07.990846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:07.990847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:07.991026 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:07.991135 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:07.991302 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:29:07.991453 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:29:07.994202 25962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:29:07.997198 25962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:29:08.010995 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.005826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.15.93.130:35967)
0927 16:29:08.006144 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.006156 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.006175 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:29:08.006252 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.006260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.006470 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.006611 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:08.006948 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008192 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.009768 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.010313 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.010514 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.010666 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.010757 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.010843 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":105,"threads_started":1}
I20260927 16:29:08.010986 26433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.005043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35967 (local address 127.0.0.1:45646)
0927 16:29:08.005441 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.005729 (+ 288us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:08.005848 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:08.006638 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:08.006644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:08.006670 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:08.006810 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.006820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008336 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.008339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.009648 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.009657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.010480 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.010485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.010584 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.010618 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:29:08.010721 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:29:08.010833 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":91,"thread_start_us":83,"threads_started":1}
I20260927 16:29:08.011269 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.005757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:46205 (local address 127.0.0.1:50496)
0927 16:29:08.006794 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.006807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:08.006906 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:08.007310 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:08.007313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:08.007330 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:08.007451 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.007458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008425 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.008430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.009653 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.009662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.010894 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.010897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.010989 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.011013 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:29:08.011077 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 16:29:08.011140 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":74,"thread_start_us":87,"threads_started":1}
I20260927 16:29:08.011689 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.006162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.15.93.131:46205)
0927 16:29:08.006658 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.006670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.006915 (+ 245us) server_negotiation.cc:407] Connection header received
0927 16:29:08.007001 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.007008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.007175 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.007324 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:08.007565 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008305 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.009799 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.010596 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.010703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.010832 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.011445 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.011538 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":96,"threads_started":1}
I20260927 16:29:08.013998 26355 tablet_service.cc:1467] Processing CreateTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf (DEFAULT_TABLE table=test-table [id=cfbe2efa60174e2cb348722949c3a6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:08.013998 26221 tablet_service.cc:1467] Processing CreateTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf (DEFAULT_TABLE table=test-table [id=cfbe2efa60174e2cb348722949c3a6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:08.014878 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b97fdeb3d24f9f82fcee458edcbccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:08.014886 26355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b97fdeb3d24f9f82fcee458edcbccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:08.016359 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.005295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.15.93.129:43377)
0927 16:29:08.005717 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.005731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.006096 (+ 365us) server_negotiation.cc:407] Connection header received
0927 16:29:08.006321 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.006326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.006470 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.006598 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:08.007051 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008192 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.010909 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.011790 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.011875 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.012012 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.016094 (+ 4082us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.016210 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":97,"threads_started":1}
I20260927 16:29:08.016791 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.005067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43377 (local address 127.0.0.1:50200)
0927 16:29:08.006003 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.006017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:08.006181 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:08.006814 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:08.006816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:08.006831 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:08.006951 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.006958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.008367 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.008371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.010776 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.010786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.015843 (+ 5057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.015847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.015937 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.015958 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:29:08.016567 (+ 609us) client_negotiation.cc:240] Negotiation successful
0927 16:29:08.016661 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":108,"thread_start_us":98,"threads_started":1}
I20260927 16:29:08.019754 26087 tablet_service.cc:1467] Processing CreateTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf (DEFAULT_TABLE table=test-table [id=cfbe2efa60174e2cb348722949c3a6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:08.020830 26087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b97fdeb3d24f9f82fcee458edcbccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:08.021431 26438 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Bootstrap starting.
I20260927 16:29:08.021917 26439 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Bootstrap starting.
I20260927 16:29:08.024080 26439 tablet_bootstrap.cc:654] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:08.024271 26438 tablet_bootstrap.cc:654] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:08.025183 26438 log.cc:824] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:08.025207 26439 log.cc:824] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:08.027098 26442 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Bootstrap starting.
I20260927 16:29:08.027163 26438 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: No bootstrap required, opened a new log
I20260927 16:29:08.027148 26439 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: No bootstrap required, opened a new log
I20260927 16:29:08.027338 26438 ts_tablet_manager.cc:1397] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:29:08.027520 26439 ts_tablet_manager.cc:1397] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:29:08.029209 26442 tablet_bootstrap.cc:654] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:08.030413 26442 log.cc:824] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:08.030387 26439 raft_consensus.cc:348] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.030661 26439 raft_consensus.cc:374] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:08.030714 26439 raft_consensus.cc:729] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4fe5450e804dd6a3cc43b65856bffb, State: Initialized, Role: FOLLOWER
I20260927 16:29:08.031172 26439 consensus_queue.cc:260] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.031242 26438 raft_consensus.cc:348] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.031497 26438 raft_consensus.cc:374] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:08.031555 26438 raft_consensus.cc:729] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e415e7d2db464d3da9ac141d1a944345, State: Initialized, Role: FOLLOWER
I20260927 16:29:08.031895 26421 heartbeater.cc:503] Master 127.15.93.190:34961 was elected leader, sending a full tablet report...
I20260927 16:29:08.031960 26438 consensus_queue.cc:260] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.032138 26439 ts_tablet_manager.cc:1428] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260927 16:29:08.032384 26442 tablet_bootstrap.cc:492] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: No bootstrap required, opened a new log
I20260927 16:29:08.032537 26442 ts_tablet_manager.cc:1397] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:29:08.032665 26287 heartbeater.cc:503] Master 127.15.93.190:34961 was elected leader, sending a full tablet report...
I20260927 16:29:08.034017 26438 ts_tablet_manager.cc:1428] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:29:08.037230 26442 raft_consensus.cc:348] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.037408 26442 raft_consensus.cc:374] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:08.037438 26442 raft_consensus.cc:729] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edccadedb14443b8facfcce9bbb53e4, State: Initialized, Role: FOLLOWER
I20260927 16:29:08.037710 26442 consensus_queue.cc:260] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.038225 26153 heartbeater.cc:503] Master 127.15.93.190:34961 was elected leader, sending a full tablet report...
I20260927 16:29:08.038421 26442 ts_tablet_manager.cc:1428] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 16:29:08.062863 26154 tablet.cc:2369] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:29:08.080372 26445 raft_consensus.cc:482] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:29:08.080542 26445 raft_consensus.cc:504] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.081578 26445 leader_election.cc:288] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edccadedb14443b8facfcce9bbb53e4 (127.15.93.129:43377), 0e4fe5450e804dd6a3cc43b65856bffb (127.15.93.131:46205)
I20260927 16:29:08.085127 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.081832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:39747 (local address 127.15.93.129:43377)
0927 16:29:08.081956 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.081961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.081997 (+ 36us) server_negotiation.cc:407] Connection header received
0927 16:29:08.082144 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.082147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.082192 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.082273 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:08.082548 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.083040 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.084185 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.084565 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.084693 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.084768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.084940 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.085012 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:29:08.085124 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.081775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43377 (local address 127.15.93.130:39747)
0927 16:29:08.081923 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.081940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:08.082017 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:08.082297 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:08.082300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:08.082318 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:08.082431 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.082436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.083154 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.083160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.084069 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.084078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.084681 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.084685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.084790 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.084821 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:29:08.084915 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 16:29:08.084988 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 16:29:08.085556 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.082045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:44355 (local address 127.15.93.131:46205)
0927 16:29:08.082158 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.082162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.082358 (+ 196us) server_negotiation.cc:407] Connection header received
0927 16:29:08.082504 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.082507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.082557 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.082642 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:08.082887 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.083422 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.084464 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.084977 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.085098 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.085154 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.085388 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.085455 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:29:08.085568 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.081797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:46205 (local address 127.15.93.130:44355)
0927 16:29:08.082273 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.082283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:08.082404 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:08.082654 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:08.082657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:08.082672 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:08.082783 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.082787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.083531 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.083536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.084344 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:08.084353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.085099 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.085102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:08.085205 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.085239 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:29:08.085369 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 16:29:08.085435 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":127,"threads_started":1}
I20260927 16:29:08.085848 26107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b97fdeb3d24f9f82fcee458edcbccf" candidate_uuid: "e415e7d2db464d3da9ac141d1a944345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edccadedb14443b8facfcce9bbb53e4" is_pre_election: true
I20260927 16:29:08.086165 26107 raft_consensus.cc:2454] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e415e7d2db464d3da9ac141d1a944345 in term 0.
I20260927 16:29:08.086397 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b97fdeb3d24f9f82fcee458edcbccf" candidate_uuid: "e415e7d2db464d3da9ac141d1a944345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" is_pre_election: true
I20260927 16:29:08.086687 26375 raft_consensus.cc:2454] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e415e7d2db464d3da9ac141d1a944345 in term 0.
I20260927 16:29:08.086660 26177 leader_election.cc:302] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edccadedb14443b8facfcce9bbb53e4, e415e7d2db464d3da9ac141d1a944345; no voters:
I20260927 16:29:08.086954 26445 raft_consensus.cc:2781] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:29:08.087051 26445 raft_consensus.cc:482] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:29:08.087110 26445 raft_consensus.cc:3037] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:08.088131 26445 raft_consensus.cc:504] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.088479 26445 leader_election.cc:288] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [CANDIDATE]: Term 1 election: Requested vote from peers 6edccadedb14443b8facfcce9bbb53e4 (127.15.93.129:43377), 0e4fe5450e804dd6a3cc43b65856bffb (127.15.93.131:46205)
I20260927 16:29:08.088804 26107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b97fdeb3d24f9f82fcee458edcbccf" candidate_uuid: "e415e7d2db464d3da9ac141d1a944345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edccadedb14443b8facfcce9bbb53e4"
I20260927 16:29:08.088845 26375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b97fdeb3d24f9f82fcee458edcbccf" candidate_uuid: "e415e7d2db464d3da9ac141d1a944345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e4fe5450e804dd6a3cc43b65856bffb"
I20260927 16:29:08.088937 26107 raft_consensus.cc:3037] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:08.088964 26375 raft_consensus.cc:3037] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:08.089892 26107 raft_consensus.cc:2454] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e415e7d2db464d3da9ac141d1a944345 in term 1.
I20260927 16:29:08.089977 26375 raft_consensus.cc:2454] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e415e7d2db464d3da9ac141d1a944345 in term 1.
I20260927 16:29:08.090149 26177 leader_election.cc:302] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edccadedb14443b8facfcce9bbb53e4, e415e7d2db464d3da9ac141d1a944345; no voters:
I20260927 16:29:08.090355 26445 raft_consensus.cc:2781] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:08.090584 26445 raft_consensus.cc:686] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 1 LEADER]: Becoming Leader. State: Replica: e415e7d2db464d3da9ac141d1a944345, State: Running, Role: LEADER
I20260927 16:29:08.090875 26445 consensus_queue.cc:237] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } }
I20260927 16:29:08.093811 25962 catalog_manager.cc:5495] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 reported cstate change: term changed from 0 to 1, leader changed from <none> to e415e7d2db464d3da9ac141d1a944345 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "e415e7d2db464d3da9ac141d1a944345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e415e7d2db464d3da9ac141d1a944345" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 } health_report { overall_health: UNKNOWN } } }
I20260927 16:29:08.106307 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.103084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.15.93.130:35967)
0927 16:29:08.103205 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:08.103209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:08.103267 (+ 58us) server_negotiation.cc:407] Connection header received
0927 16:29:08.103411 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:08.103413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:08.103460 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:08.103540 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:29:08.103759 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.104218 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.105001 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:08.105200 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:08.105293 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:08.105529 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:29:08.105919 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:29:08.106006 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:08.106097 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 16:29:08.106141 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:29:08.111630 26107 raft_consensus.cc:1261] T 56b97fdeb3d24f9f82fcee458edcbccf P 6edccadedb14443b8facfcce9bbb53e4 [term 1 FOLLOWER]: Refusing update from remote peer e415e7d2db464d3da9ac141d1a944345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:29:08.111632 26375 raft_consensus.cc:1261] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 1 FOLLOWER]: Refusing update from remote peer e415e7d2db464d3da9ac141d1a944345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:29:08.112380 26448 consensus_queue.cc:1035] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edccadedb14443b8facfcce9bbb53e4" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:29:08.112562 26445 consensus_queue.cc:1035] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e4fe5450e804dd6a3cc43b65856bffb" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:29:08.120260 26451 mvcc.cc:205] Tried to move back new op lower bound from 7333996741058494464 to 7333996740982837248. Current Snapshot: MvccSnapshot[applied={T|T < 7333996741058494464}]
I20260927 16:29:08.120779 26452 mvcc.cc:205] Tried to move back new op lower bound from 7333996741058494464 to 7333996740982837248. Current Snapshot: MvccSnapshot[applied={T|T < 7333996741058494464}]
I20260927 16:29:08.128464 25961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:29:08.128646 25961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:29:08.130993 25961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1fa6eb0d7da44ca964170955774cc10: Sending DeleteTablet for 3 replicas of tablet 56b97fdeb3d24f9f82fcee458edcbccf
I20260927 16:29:08.131699 26221 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:08 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:45646
I20260927 16:29:08.131923 26355 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:08 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:50496
I20260927 16:29:08.132143 26087 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b97fdeb3d24f9f82fcee458edcbccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:08 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:50200
I20260927 16:29:08.132378 26462 tablet_replica.cc:330] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: stopping tablet replica
I20260927 16:29:08.132496 26461 tablet_replica.cc:330] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: stopping tablet replica
I20260927 16:29:08.132747 26462 raft_consensus.cc:2229] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:29:08.132747 26461 raft_consensus.cc:2229] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:29:08.132900 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 26024
I20260927 16:29:08.133610 26461 raft_consensus.cc:2258] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:29:08.133843 26462 raft_consensus.cc:2258] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:29:08.135558 26462 ts_tablet_manager.cc:1905] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:29:08.135558 26461 ts_tablet_manager.cc:1905] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:29:08.138120 26462 ts_tablet_manager.cc:1918] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:29:08.138115 26461 ts_tablet_manager.cc:1918] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:29:08.138211 26461 log.cc:1196] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/56b97fdeb3d24f9f82fcee458edcbccf
I20260927 16:29:08.138320 26462 log.cc:1196] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/56b97fdeb3d24f9f82fcee458edcbccf
I20260927 16:29:08.138540 26461 ts_tablet_manager.cc:1939] T 56b97fdeb3d24f9f82fcee458edcbccf P e415e7d2db464d3da9ac141d1a944345: Deleting consensus metadata
I20260927 16:29:08.138835 26462 ts_tablet_manager.cc:1939] T 56b97fdeb3d24f9f82fcee458edcbccf P 0e4fe5450e804dd6a3cc43b65856bffb: Deleting consensus metadata
I20260927 16:29:08.139753 25946 catalog_manager.cc:4841] TS e415e7d2db464d3da9ac141d1a944345 (127.15.93.130:35967): tablet 56b97fdeb3d24f9f82fcee458edcbccf (table test-table [id=cfbe2efa60174e2cb348722949c3a6c6]) successfully deleted
I20260927 16:29:08.139928 25948 catalog_manager.cc:4841] TS 0e4fe5450e804dd6a3cc43b65856bffb (127.15.93.131:46205): tablet 56b97fdeb3d24f9f82fcee458edcbccf (table test-table [id=cfbe2efa60174e2cb348722949c3a6c6]) successfully deleted
W20260927 16:29:08.142421 25949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:43377 (error 108)
I20260927 16:29:08.143203 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 26158
I20260927 16:29:08.144156 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:08.143496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43377 (local address 127.0.0.1:50208)
0927 16:29:08.144008 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:08.144076 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260927 16:29:08.144420 25949 catalog_manager.cc:4584] TS 6edccadedb14443b8facfcce9bbb53e4 (127.15.93.129:43377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56b97fdeb3d24f9f82fcee458edcbccf: Network error: Client connection negotiation failed: client connection to 127.15.93.129:43377: connect: Connection refused (error 111)
I20260927 16:29:08.149715 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 26292
I20260927 16:29:08.154253 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 25930
2026-09-27T16:29:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2184 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:29:08 dist-test-slave-grzf krb5kdc[26467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:29:08 dist-test-slave-grzf krb5kdc[26467](info): set up 2 sockets
Sep 27 16:29:08 dist-test-slave-grzf krb5kdc[26467](info): commencing operation
krb5kdc: starting...
W20260927 16:29:09.180498 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:29:09 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:29:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangeradmin_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangeradmin_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangerlookup_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangerlookup_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
I20260927 16:29:09.340370 15734 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5Uwdq0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/postgres -l logfile start
2026-09-27 16:29:11.701 UTC [26497] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:29:11.701 UTC [26497] LOG: listening on IPv4 address "127.15.93.148", port 59003
2026-09-27 16:29:11.705 UTC [26497] LOG: listening on Unix socket "/tmp/.s.PGSQL.59003"
2026-09-27 16:29:11.738 UTC [26500] LOG: database system was shut down at 2026-09-27 16:29:10 UTC
2026-09-27 16:29:11.743 UTC [26497] LOG: database system is ready to accept connections
I20260927 16:29:12.688150 15734 mini_postgres.cc:96] Postgres bound to 59003
2026-09-27 16:29:12.694 UTC [26508] FATAL: database "slave" does not exist
127.15.93.148:59003 - accepting connections
I20260927 16:29:12.694996 15734 mini_ranger.cc:161] Starting Ranger...
I20260927 16:29:12.709278 15734 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:29:13.159250 15734 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:29:13.159354 15734 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin
2026-09-27 16:29:13,446 [I] DB FLAVOR :POSTGRES
2026-09-27 16:29:13,446 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:29:13,446 [I] Checking connection..
2026-09-27 16:29:13,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:29:13,685 [I] Checking connection passed.
2026-09-27 16:29:13,685 [I] --------- Verifying version history table ---------
2026-09-27 16:29:13,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:13,919 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:29:13,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:14,152 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:29:14,153 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:29:14,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:29:14.402 UTC [26572] WARNING: there is no transaction in progress
2026-09-27 16:29:14,409 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:29:14,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:14,657 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:29:14,657 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:29:14,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:29:14,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:29:15,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grzf.c.gcp-upstream.internal','N') ;"
2026-09-27 16:29:15,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:29:15,601 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:29:15,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:29:15,859 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:29:15,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:29:16,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:29:16,324 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:29:16,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:29:17.270 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.286 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.301 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.316 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.334 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.354 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.412 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.418 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.432 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.441 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.451 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.464 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.473 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.479 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.485 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:17.491 UTC [26680] WARNING: there is no transaction in progress
2026-09-27 16:29:18,082 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:29:18,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:29:18,303 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:29:18,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:29:18,527 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:29:18,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:29:18,782 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:29:18,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:29:19,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:29:19,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grzf.c.gcp-upstream.internal';"
2026-09-27 16:29:19,476 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:29:19,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:29:19,701 [I] DB_PATCHES have already been applied
I20260927 16:29:19.707259 15734 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5Uwdq0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:29:19.707320 15734 mini_ranger.cc:193] Using host: 127.15.93.148
I20260927 16:29:19.710572 15734 mini_ranger.cc:227] Ranger admin URL: http://127.15.93.148:44399
Sep 27, 2026 4:29:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:29:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:29:20.818220 15734 mini_ranger.cc:160] Time spent starting Ranger: real 8.123s user 0.006s sys 0.003s
Sep 27, 2026 4:29:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:29:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 4:29:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:29:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:29:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:29:41.749570 15734 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangerkms_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/rangerkms_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.93.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.93.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:29:41 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526581, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:29:41.837677 15734 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:29:41.847366 15734 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:29:42.632957 15734 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:29:42.633054 15734 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms
I20260927 16:29:42.633159 15734 mini_ranger_kms.cc:227] Using postgres at 127.15.93.148:59003
2026-09-27 16:29:42,807 [I] DB FLAVOR :POSTGRES
2026-09-27 16:29:42,807 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:29:42,807 [I] Checking connection
2026-09-27 16:29:42,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:29:43,043 [I] connection success
2026-09-27 16:29:43,043 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:29:43,043 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:29:43,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:29:43,272 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:29:43,272 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:29:43,273 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:29:43,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:59003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:29:43,542 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:29:43.981673 15734 mini_ranger_kms.cc:310] Created kms service
I20260927 16:29:44.147116 15734 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:29:44.190661 15734 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:29:44 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526581, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:29:44.581288 15734 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:29:44.581439 15734 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms:/tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:29:44.581470 15734 mini_ranger_kms.cc:240] Using host: 127.15.93.148
I20260927 16:29:44.584924 15734 mini_ranger_kms.cc:276] Ranger KMS PID: 26860
I20260927 16:29:44.585081 15734 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.93.148:34451
Sep 27, 2026 4:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:29:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:29:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 16:29:46 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:46.850075 15734 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.012s user 0.021s sys 0.137s
I20260927 16:29:46.850210 15734 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:29:46.850335 15734 mini_ranger_kms.cc:401] 127.15.93.148:34451/kms/v1/keys
Sep 27 16:29:47 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:29:47 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:29:49 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:29:49.860487 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40733
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:40733
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:34451/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:49.980919 26911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:49.981169 26911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:49.981210 26911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:49.985190 26911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:29:49.985239 26911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:49.985258 26911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:49.985303 26911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:29:49.985333 26911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:29:49.990872 26911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.93.148:34451/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:40733
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:49.991917 26911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:49.992892 26911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:50.001116 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:50.001159 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:50.001302 26911 server_base.cc:1034] running on GCE node
W20260927 16:29:50.001145 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:50.001983 26911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:50.003062 26911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:50.004233 26911 hybrid_clock.cc:648] HybridClock initialized: now 1790526590004215 us; error 43 us; skew 500 ppm
Sep 27 16:29:50 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:50.007133 26911 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:29:50.008176 26911 webserver.cc:460] Webserver started at http://127.15.93.190:44111/ using document root <none> and password file <none>
I20260927 16:29:50.008836 26911 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:50.008912 26911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:50.009130 26911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:29:50 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526590, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:29:50.062853 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "bed5576d1efc4625878600405b5eb2dc"
format_stamp: "Formatted at 2026-09-27 16:29:50 on dist-test-slave-grzf"
server_key: "b7fe530412bb53344cf7bfc1d48fba7b"
server_key_iv: "14e2325865820a9f1df1c701b3998953"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:50.063556 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "bed5576d1efc4625878600405b5eb2dc"
format_stamp: "Formatted at 2026-09-27 16:29:50 on dist-test-slave-grzf"
server_key: "b7fe530412bb53344cf7bfc1d48fba7b"
server_key_iv: "14e2325865820a9f1df1c701b3998953"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:50.067973 26911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:29:50.083777 26926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:50.085240 26911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:29:50.085383 26911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "bed5576d1efc4625878600405b5eb2dc"
format_stamp: "Formatted at 2026-09-27 16:29:50 on dist-test-slave-grzf"
server_key: "b7fe530412bb53344cf7bfc1d48fba7b"
server_key_iv: "14e2325865820a9f1df1c701b3998953"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:50.085520 26911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:50.117158 26911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:50.120146 26911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:50.120337 26911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:50.128326 26911 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:40733
I20260927 16:29:50.128353 26978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:40733 every 8 connection(s)
I20260927 16:29:50.129235 26911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:29:50.132579 26979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:50.138370 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 26911
I20260927 16:29:50.138506 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:29:50.138780 15734 external_mini_cluster.cc:1428] Setting key 9dd4792e3891791e66dd95ebfea59051
I20260927 16:29:50.139556 26979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: Bootstrap starting.
I20260927 16:29:50.142351 26979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:50.143195 26979 log.cc:824] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:50.145498 26979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: No bootstrap required, opened a new log
I20260927 16:29:50.150141 26979 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } }
I20260927 16:29:50.150485 26979 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:50.150557 26979 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed5576d1efc4625878600405b5eb2dc, State: Initialized, Role: FOLLOWER
I20260927 16:29:50.150997 26979 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } }
I20260927 16:29:50.151350 26979 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:29:50.151443 26979 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:29:50.151542 26979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:29:50 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:50.152614 26979 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } }
I20260927 16:29:50.152951 26979 leader_election.cc:302] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed5576d1efc4625878600405b5eb2dc; no voters:
I20260927 16:29:50.153282 26979 leader_election.cc:288] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:29:50.153388 26984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:50.153609 26984 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [term 1 LEADER]: Becoming Leader. State: Replica: bed5576d1efc4625878600405b5eb2dc, State: Running, Role: LEADER
I20260927 16:29:50.154213 26984 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } }
I20260927 16:29:50.154783 26979 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:29:50.155474 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed5576d1efc4625878600405b5eb2dc. Latest consensus state: current_term: 1 leader_uuid: "bed5576d1efc4625878600405b5eb2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } } }
I20260927 16:29:50.155589 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:50.156096 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed5576d1efc4625878600405b5eb2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed5576d1efc4625878600405b5eb2dc" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40733 } } }
I20260927 16:29:50.156248 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc [sys.catalog]: This master's current role is: LEADER
I20260927 16:29:50.157124 26979 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:29:50.160517 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:50.140193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.15.93.190:40733)
0927 16:29:50.141232 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:50.141281 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:50.141330 (+ 49us) server_negotiation.cc:407] Connection header received
0927 16:29:50.142820 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:50.142849 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:50.143565 (+ 716us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:50.145753 (+ 2188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:50.146428 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:50.148402 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:50.149272 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:50.149747 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:50.149912 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:50.152648 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:50.152653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:50.152665 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:50.152708 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:50.154833 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:50.155443 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:50.155448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:50.155454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:50.155588 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:50.156004 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:50.156009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:50.156012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:50.156519 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:50.156707 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:50.157576 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 16:29:50.157861 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":129,"threads_started":1}
I20260927 16:29:50.164018 26990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:29:50.167259 26990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:29:50.177176 26990 catalog_manager.cc:1347] Generated new cluster ID: 64b80d8b919f4adb8d8035371491ff95
I20260927 16:29:50.177244 26990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:29:50.196439 26990 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:29:50.198879 26990 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:29:50.204005 26990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: Generated new TSK 0
I20260927 16:29:50.204756 26990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:29:50.911221 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:34451/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:51.044471 27024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:51.044770 27024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:51.044894 27024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:51.051301 27024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:51.051405 27024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:51.051510 27024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:29:51.059736 27024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.93.148:34451/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:51.061192 27024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:51.062577 27024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:51.074083 27032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:51.074214 27031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:51.074535 27034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:51.075472 27024 server_base.cc:1034] running on GCE node
I20260927 16:29:51.075928 27024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:51.076680 27024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:51.077917 27024 hybrid_clock.cc:648] HybridClock initialized: now 1790526591077897 us; error 35 us; skew 500 ppm
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:51.085038 27024 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:29:51.086222 27024 webserver.cc:460] Webserver started at http://127.15.93.129:46637/ using document root <none> and password file <none>
I20260927 16:29:51.087010 27024 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:51.087154 27024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:51.087482 27024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:29:51.116714 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "ffe51d2a05594c3f95f4cc823bf69a5a"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "92c2b019d2e7e24a5e100eda009f280b"
server_key_iv: "611818cb6eb113986630b82fba98b2a5"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.117439 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "ffe51d2a05594c3f95f4cc823bf69a5a"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "92c2b019d2e7e24a5e100eda009f280b"
server_key_iv: "611818cb6eb113986630b82fba98b2a5"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.123015 27024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 16:29:51.141522 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.143115 27024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 16:29:51.143289 27024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "ffe51d2a05594c3f95f4cc823bf69a5a"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "92c2b019d2e7e24a5e100eda009f280b"
server_key_iv: "611818cb6eb113986630b82fba98b2a5"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.143507 27024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:51.161772 27024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:51.165518 27024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:51.165766 27024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:51.166481 27024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:51.167563 27024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:51.167680 27024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.167800 27024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:51.167876 27024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.180214 27024 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:34437
I20260927 16:29:51.181370 27024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:29:51.182299 27156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:34437 every 8 connection(s)
I20260927 16:29:51.182732 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 27024
I20260927 16:29:51.182852 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:29:51.183151 15734 external_mini_cluster.cc:1428] Setting key b8e89a33f8cdc860743a24f02ab50221
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:51.205588 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.184041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:40733 (local address 127.15.93.129:59149)
0927 16:29:51.187887 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.187963 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.189196 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.190094 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.190106 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.190482 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:51.191143 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.191163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.192428 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.192431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.193898 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.193909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.194661 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.194665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.194875 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.195590 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:51.195611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:51.199275 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:51.202094 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.202099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.202116 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.202466 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.202806 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.202809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.202811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.203008 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.203562 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:51.203569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:51.203980 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.204203 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.204549 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3612,"thread_start_us":127,"threads_started":1}
I20260927 16:29:51.206334 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.183582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:59149 (local address 127.15.93.190:40733)
0927 16:29:51.187598 (+ 4016us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.187608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.188598 (+ 990us) server_negotiation.cc:407] Connection header received
0927 16:29:51.189527 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.189530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.189586 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.189760 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:51.191314 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.192284 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.194082 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.194473 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.194592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.199526 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:51.199529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:51.199532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:51.199562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:51.201953 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.202591 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.202595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.202597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.202694 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.203124 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.203128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.203130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.203399 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:51.203503 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.206021 (+ 2518us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.206154 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3928,"thread_start_us":89,"threads_started":1}
I20260927 16:29:51.208165 27157 heartbeater.cc:347] Connected to a master server at 127.15.93.190:40733
I20260927 16:29:51.208411 27157 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:51.208997 27157 heartbeater.cc:511] Master 127.15.93.190:40733 requested a full tablet report, sending...
I20260927 16:29:51.210731 26943 ts_manager.cc:194] Registered new tserver with Master: ffe51d2a05594c3f95f4cc823bf69a5a (127.15.93.129:34437)
I20260927 16:29:51.212283 26943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:59149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:29:51.216674 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:34451/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 16:29:51.342821 27163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:51.343118 27163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:51.343189 27163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:51.347672 27163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:51.347721 27163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:51.347806 27163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:29:51.353389 27163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.93.148:34451/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:51.354553 27163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:51.355724 27163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:51.366299 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:51.366631 27163 server_base.cc:1034] running on GCE node
W20260927 16:29:51.367146 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:51.367333 27172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:51.368012 27163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:51.368780 27163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:51.369984 27163 hybrid_clock.cc:648] HybridClock initialized: now 1790526591369962 us; error 36 us; skew 500 ppm
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:51.373520 27163 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:29:51.374514 27163 webserver.cc:460] Webserver started at http://127.15.93.130:35131/ using document root <none> and password file <none>
I20260927 16:29:51.375234 27163 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:51.375360 27163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:51.375636 27163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:29:51.392848 27163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "c17dc6efc85f476fbea3761e27833bad"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "571dc4bf0cd99675598d7e373b85d017"
server_key_iv: "978551377ea2195532380b9dbe2bd9a7"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.394105 27163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "c17dc6efc85f476fbea3761e27833bad"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "571dc4bf0cd99675598d7e373b85d017"
server_key_iv: "978551377ea2195532380b9dbe2bd9a7"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.401031 27163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 16:29:51.419739 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.421391 27163 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 16:29:51.421535 27163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "c17dc6efc85f476fbea3761e27833bad"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "571dc4bf0cd99675598d7e373b85d017"
server_key_iv: "978551377ea2195532380b9dbe2bd9a7"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.421674 27163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:51.443699 27163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:51.447275 27163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:51.447554 27163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:51.448348 27163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:51.449436 27163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:51.449555 27163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.449661 27163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:51.449733 27163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.461010 27163 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:32983
I20260927 16:29:51.461069 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:32983 every 8 connection(s)
I20260927 16:29:51.462237 27163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:29:51.467433 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 27163
I20260927 16:29:51.467568 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:29:51.467885 15734 external_mini_cluster.cc:1428] Setting key 7d37ee9526f3bc5f73a7541d11affa3d
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:51.482367 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.464177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55047 (local address 127.15.93.190:40733)
0927 16:29:51.464316 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.464321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.466398 (+ 2077us) server_negotiation.cc:407] Connection header received
0927 16:29:51.468872 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.468876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.468929 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.468982 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:51.470522 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.471361 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.473277 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.473638 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.473716 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.478163 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:51.478167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:51.478170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:51.478196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:51.479959 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.480603 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.480607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.480609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.480699 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.481112 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.481116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.481118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.481337 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:51.481408 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.482065 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.482197 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:29:51.484193 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.464640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:40733 (local address 127.15.93.130:55047)
0927 16:29:51.466220 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.466294 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.468622 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.469371 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.469379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.469727 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:51.470358 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.470376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.471534 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.471538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.473141 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.473152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.473906 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.473910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.474100 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.475799 (+ 1699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:51.475819 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:51.477971 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:51.480085 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.480093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.480111 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.480475 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.480814 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.480818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.480821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.481001 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.481488 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:51.481495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:51.481795 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.482811 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.483134 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":129,"threads_started":1}
I20260927 16:29:51.485659 27293 heartbeater.cc:347] Connected to a master server at 127.15.93.190:40733
I20260927 16:29:51.485889 27293 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:51.486426 27293 heartbeater.cc:511] Master 127.15.93.190:40733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:29:51.487550 26943 ts_manager.cc:194] Registered new tserver with Master: c17dc6efc85f476fbea3761e27833bad (127.15.93.130:32983)
I20260927 16:29:51.488278 26943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:55047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:29:51.504652 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:34451/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:37607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:29:51.634079 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:29:51.634357 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:29:51.634464 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:29:51.638623 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:29:51.638705 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:29:51.638803 27298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:29:51.645957 27298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:37607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.93.148:34451/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:40733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:29:51.647082 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:29:51.648236 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:29:51.659971 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:51.660075 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:29:51.660205 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:29:51.661326 27298 server_base.cc:1034] running on GCE node
I20260927 16:29:51.661811 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:29:51.662531 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:29:51.663708 27298 hybrid_clock.cc:648] HybridClock initialized: now 1790526591663638 us; error 83 us; skew 500 ppm
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:29:51.667219 27298 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:29:51.668316 27298 webserver.cc:460] Webserver started at http://127.15.93.131:39377/ using document root <none> and password file <none>
I20260927 16:29:51.669006 27298 fs_manager.cc:359] Metadata directory not provided
I20260927 16:29:51.669101 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:29:51.669332 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:29:51.690306 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "08d2759c8a7f47e099641c26b7867c71"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "b206a51195799d96b55c07b1a149ba4c"
server_key_iv: "275d074d5c2b89dbf24e766d46808766"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.691107 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "08d2759c8a7f47e099641c26b7867c71"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "b206a51195799d96b55c07b1a149ba4c"
server_key_iv: "275d074d5c2b89dbf24e766d46808766"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.697342 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260927 16:29:51.723764 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.726742 27298 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 16:29:51.726971 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "08d2759c8a7f47e099641c26b7867c71"
format_stamp: "Formatted at 2026-09-27 16:29:51 on dist-test-slave-grzf"
server_key: "b206a51195799d96b55c07b1a149ba4c"
server_key_iv: "275d074d5c2b89dbf24e766d46808766"
server_key_version: "kuduclusterkey@0"
I20260927 16:29:51.727128 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:29:51.775856 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:29:51.779497 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:29:51.779680 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:29:51.780364 27298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:29:51.781392 27298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:29:51.781456 27298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.781509 27298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:29:51.781543 27298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:29:51.792397 27298 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:35631
I20260927 16:29:51.792548 27426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:35631 every 8 connection(s)
I20260927 16:29:51.793475 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:29:51.802366 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 27298
I20260927 16:29:51.802489 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:29:51.802809 15734 external_mini_cluster.cc:1428] Setting key 982c8f3bbf53b7bc9f762d9b8b639066
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:29:51.807336 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.795446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:40721 (local address 127.15.93.190:40733)
0927 16:29:51.795577 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.795581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.796455 (+ 874us) server_negotiation.cc:407] Connection header received
0927 16:29:51.797419 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.797423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.797480 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.797590 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:51.798631 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.799154 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.800680 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.800904 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.801017 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.803616 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:51.803619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:51.803622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:51.803648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:51.805355 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.805838 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.805842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.805843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.805925 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.806265 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.806268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.806270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.806444 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:51.806548 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.807058 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.807172 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:29:51.808296 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.795855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:40733 (local address 127.15.93.131:40721)
0927 16:29:51.796308 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.796358 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.797212 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.797771 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.797778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.798034 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:29:51.798481 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.798494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.799277 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.799280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.800554 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.800564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.801047 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.801051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.801233 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.801818 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:29:51.801833 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:29:51.803439 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:29:51.805469 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.805475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.805488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.805735 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.806019 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:29:51.806022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:29:51.806024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:29:51.806165 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:29:51.806544 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:29:51.806550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:29:51.806783 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.806996 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.807293 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":144,"threads_started":1}
I20260927 16:29:51.809327 27427 heartbeater.cc:347] Connected to a master server at 127.15.93.190:40733
I20260927 16:29:51.809546 27427 heartbeater.cc:464] Registering TS with master...
I20260927 16:29:51.810009 27427 heartbeater.cc:511] Master 127.15.93.190:40733 requested a full tablet report, sending...
I20260927 16:29:51.811031 26943 ts_manager.cc:194] Registered new tserver with Master: 08d2759c8a7f47e099641c26b7867c71 (127.15.93.131:35631)
I20260927 16:29:51.811645 26943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:40721
I20260927 16:29:51.816654 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:29:51.826236 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.818662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.15.93.190:40733)
0927 16:29:51.818785 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.818789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.819046 (+ 257us) server_negotiation.cc:407] Connection header received
0927 16:29:51.819165 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.819168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.819209 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.819301 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:29:51.820082 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.820587 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.821408 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.821635 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.821742 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.823078 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:29:51.823082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:29:51.823085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:29:51.823116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:29:51.824611 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.825023 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.825028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.825030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.825111 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:29:51.825387 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:29:51.825392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:29:51.825394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:29:51.825555 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:29:51.825694 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.825927 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.826036 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:29:51.829285 26943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:29:51.832403 26943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:29:51.857764 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.847670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:34437 (local address 127.0.0.1:57756)
0927 16:29:51.849854 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.849865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.849931 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.850786 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.850792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.850820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:51.851063 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.851070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.853503 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.853508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.854969 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.854979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.857213 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.857217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.857341 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.857386 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.857473 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.857608 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2127,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260927 16:29:51.858556 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.849573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.15.93.129:34437)
0927 16:29:51.850190 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.850197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.850221 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:29:51.850310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.850315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.850579 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.850745 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:51.851275 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.853371 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.856171 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.857075 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.858116 (+ 1041us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.858268 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.858322 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.858420 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":90,"threads_started":1}
I20260927 16:29:51.861477 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.847653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:35631 (local address 127.0.0.1:38700)
0927 16:29:51.849465 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.849519 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.849644 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.851756 (+ 2112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.851759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.851791 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:51.851899 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.851906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.858692 (+ 6786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.858696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.859906 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.859917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.861064 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.861068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.861175 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.861200 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.861252 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.861320 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"thread_start_us":123,"threads_started":1}
I20260927 16:29:51.862136 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.848715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.15.93.131:35631)
0927 16:29:51.851223 (+ 2508us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.851233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.851273 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:29:51.851352 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.851357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.851600 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.852064 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:51.852203 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.853423 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.860088 (+ 6665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.860909 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.861668 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.861819 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.861875 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.861972 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2416,"thread_start_us":142,"threads_started":1}
I20260927 16:29:51.861724 27073 tablet_service.cc:1467] Processing CreateTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef (DEFAULT_TABLE table=test-table [id=9294af76bbab4110922c3593ce1b4ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:51.862802 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.848754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:32983 (local address 127.0.0.1:41528)
0927 16:29:51.853646 (+ 4892us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:51.853659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:51.853740 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:51.854372 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:51.854375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:51.854392 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:51.854525 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.854531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.856026 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.856029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.857306 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:51.857318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.862360 (+ 5042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.862364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:51.862479 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.862505 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 16:29:51.862586 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:29:51.862656 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4829,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260927 16:29:51.863247 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:51.849209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.15.93.130:32983)
0927 16:29:51.851039 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:51.851049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:51.853831 (+ 2782us) server_negotiation.cc:407] Connection header received
0927 16:29:51.853907 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:51.853912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:51.854092 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:51.854238 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:51.854659 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.855890 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.857489 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:51.858327 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:51.858433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:51.858594 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:51.862983 (+ 4389us) server_negotiation.cc:299] Negotiation successful
0927 16:29:51.863101 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":129,"threads_started":1}
I20260927 16:29:51.866569 27343 tablet_service.cc:1467] Processing CreateTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef (DEFAULT_TABLE table=test-table [id=9294af76bbab4110922c3593ce1b4ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:51.866585 27227 tablet_service.cc:1467] Processing CreateTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef (DEFAULT_TABLE table=test-table [id=9294af76bbab4110922c3593ce1b4ed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:29:51.867734 27227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570a49ae3a3f4eac9d5a857584b392ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:51.868568 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570a49ae3a3f4eac9d5a857584b392ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:51.873150 27073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570a49ae3a3f4eac9d5a857584b392ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:29:51.876358 27445 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Bootstrap starting.
I20260927 16:29:51.879164 27445 tablet_bootstrap.cc:654] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:51.880120 27445 log.cc:824] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:51.882193 27445 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: No bootstrap required, opened a new log
I20260927 16:29:51.882383 27445 ts_tablet_manager.cc:1397] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:29:51.886327 27445 raft_consensus.cc:348] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.886605 27445 raft_consensus.cc:374] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:51.886655 27445 raft_consensus.cc:729] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17dc6efc85f476fbea3761e27833bad, State: Initialized, Role: FOLLOWER
I20260927 16:29:51.887095 27445 consensus_queue.cc:260] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.887953 27447 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Bootstrap starting.
I20260927 16:29:51.888188 27445 ts_tablet_manager.cc:1428] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 16:29:51.888613 27293 heartbeater.cc:503] Master 127.15.93.190:40733 was elected leader, sending a full tablet report...
I20260927 16:29:51.889015 27448 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Bootstrap starting.
I20260927 16:29:51.890794 27447 tablet_bootstrap.cc:654] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:51.891211 27448 tablet_bootstrap.cc:654] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Neither blocks nor log segments found. Creating new log.
I20260927 16:29:51.891657 27447 log.cc:824] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:51.892115 27448 log.cc:824] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:29:51.893643 27447 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: No bootstrap required, opened a new log
I20260927 16:29:51.893818 27447 ts_tablet_manager.cc:1397] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:29:51.897399 27448 tablet_bootstrap.cc:492] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: No bootstrap required, opened a new log
I20260927 16:29:51.897579 27448 ts_tablet_manager.cc:1397] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 16:29:51.897792 27447 raft_consensus.cc:348] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.898046 27447 raft_consensus.cc:374] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:51.898097 27447 raft_consensus.cc:729] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d2759c8a7f47e099641c26b7867c71, State: Initialized, Role: FOLLOWER
I20260927 16:29:51.899559 27447 consensus_queue.cc:260] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.900624 27447 ts_tablet_manager.cc:1428] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:29:51.900998 27427 heartbeater.cc:503] Master 127.15.93.190:40733 was elected leader, sending a full tablet report...
I20260927 16:29:51.901844 27448 raft_consensus.cc:348] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.902182 27448 raft_consensus.cc:374] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:29:51.902288 27448 raft_consensus.cc:729] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe51d2a05594c3f95f4cc823bf69a5a, State: Initialized, Role: FOLLOWER
I20260927 16:29:51.902804 27448 consensus_queue.cc:260] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:51.903502 27157 heartbeater.cc:503] Master 127.15.93.190:40733 was elected leader, sending a full tablet report...
I20260927 16:29:51.903837 27448 ts_tablet_manager.cc:1428] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 16:29:51.934712 27158 tablet.cc:2369] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:29:51.966401 27294 tablet.cc:2369] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 16:29:51 dist-test-slave-grzf krb5kdc[26467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526591, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
W20260927 16:29:52.050299 27428 tablet.cc:2369] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:29:52.185025 27452 raft_consensus.cc:482] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:29:52.185212 27452 raft_consensus.cc:504] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:52.186338 27452 leader_election.cc:288] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe51d2a05594c3f95f4cc823bf69a5a (127.15.93.129:34437), c17dc6efc85f476fbea3761e27833bad (127.15.93.130:32983)
I20260927 16:29:52.191992 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.186737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:34437 (local address 127.15.93.131:55373)
0927 16:29:52.186878 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:52.186896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:52.187015 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:52.188091 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:52.188094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:52.188122 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:52.188235 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:52.188241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.189304 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.189311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:52.190560 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:52.190567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.191587 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.191591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:52.191689 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:52.191724 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 16:29:52.191789 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 16:29:52.191853 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 16:29:52.192658 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.187494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:55373 (local address 127.15.93.129:34437)
0927 16:29:52.187709 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:52.187713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:52.187726 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:29:52.187771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:52.187774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:52.187812 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:52.187883 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:52.188355 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.189120 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.190737 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.191328 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.191494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:52.191544 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:52.192463 (+ 919us) server_negotiation.cc:299] Negotiation successful
0927 16:29:52.192548 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260927 16:29:52.193035 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.187601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:54821 (local address 127.15.93.130:32983)
0927 16:29:52.188038 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:52.188042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:52.188953 (+ 911us) server_negotiation.cc:407] Connection header received
0927 16:29:52.189119 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:52.189122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:52.189179 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:52.189228 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:29:52.189625 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.190335 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.191749 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.192328 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.192430 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:52.192479 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:52.192851 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 16:29:52.192920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260927 16:29:52.193192 27110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570a49ae3a3f4eac9d5a857584b392ef" candidate_uuid: "08d2759c8a7f47e099641c26b7867c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" is_pre_election: true
I20260927 16:29:52.193368 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.186950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:32983 (local address 127.15.93.131:54821)
0927 16:29:52.188867 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:52.188881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:29:52.189010 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:29:52.189358 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:29:52.189361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:29:52.189376 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:29:52.189494 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:52.189500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.190466 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.190469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:52.191623 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:29:52.191630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.192592 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.192595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:29:52.192704 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:52.192731 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:29:52.193149 (+ 418us) client_negotiation.cc:240] Negotiation successful
0927 16:29:52.193220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1855,"mutex_wait_us":11,"thread_start_us":107,"threads_started":1}
I20260927 16:29:52.193472 27110 raft_consensus.cc:2454] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d2759c8a7f47e099641c26b7867c71 in term 0.
I20260927 16:29:52.193982 27315 leader_election.cc:302] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d2759c8a7f47e099641c26b7867c71, ffe51d2a05594c3f95f4cc823bf69a5a; no voters:
I20260927 16:29:52.194026 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570a49ae3a3f4eac9d5a857584b392ef" candidate_uuid: "08d2759c8a7f47e099641c26b7867c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17dc6efc85f476fbea3761e27833bad" is_pre_election: true
I20260927 16:29:52.194299 27235 raft_consensus.cc:2454] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d2759c8a7f47e099641c26b7867c71 in term 0.
I20260927 16:29:52.194659 27452 raft_consensus.cc:2781] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:29:52.194746 27452 raft_consensus.cc:482] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:29:52.194792 27452 raft_consensus.cc:3037] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:52.196273 27452 raft_consensus.cc:504] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:52.196621 27452 leader_election.cc:288] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [CANDIDATE]: Term 1 election: Requested vote from peers ffe51d2a05594c3f95f4cc823bf69a5a (127.15.93.129:34437), c17dc6efc85f476fbea3761e27833bad (127.15.93.130:32983)
I20260927 16:29:52.197163 27235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570a49ae3a3f4eac9d5a857584b392ef" candidate_uuid: "08d2759c8a7f47e099641c26b7867c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17dc6efc85f476fbea3761e27833bad"
I20260927 16:29:52.197281 27235 raft_consensus.cc:3037] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:52.198518 27235 raft_consensus.cc:2454] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d2759c8a7f47e099641c26b7867c71 in term 1.
I20260927 16:29:52.199071 27110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570a49ae3a3f4eac9d5a857584b392ef" candidate_uuid: "08d2759c8a7f47e099641c26b7867c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a"
I20260927 16:29:52.199189 27110 raft_consensus.cc:3037] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:29:52.200344 27110 raft_consensus.cc:2454] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d2759c8a7f47e099641c26b7867c71 in term 1.
I20260927 16:29:52.200742 27315 leader_election.cc:302] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d2759c8a7f47e099641c26b7867c71, ffe51d2a05594c3f95f4cc823bf69a5a; no voters:
I20260927 16:29:52.201038 27452 raft_consensus.cc:2781] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:29:52.201261 27452 raft_consensus.cc:686] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 1 LEADER]: Becoming Leader. State: Replica: 08d2759c8a7f47e099641c26b7867c71, State: Running, Role: LEADER
I20260927 16:29:52.201581 27452 consensus_queue.cc:237] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } }
I20260927 16:29:52.204821 26943 catalog_manager.cc:5495] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d2759c8a7f47e099641c26b7867c71 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "08d2759c8a7f47e099641c26b7867c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d2759c8a7f47e099641c26b7867c71" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 } health_report { overall_health: UNKNOWN } } }
I20260927 16:29:52.229564 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.226146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.15.93.131:35631)
0927 16:29:52.226270 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:29:52.226273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:29:52.226286 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:29:52.226420 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:29:52.226422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:29:52.226474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:29:52.226583 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:29:52.226819 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.227318 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.228276 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:29:52.228485 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:29:52.228563 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:29:52.228869 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:29:52.229252 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:29:52.229339 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:29:52.229417 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 16:29:52.229463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:29:52.235157 27110 raft_consensus.cc:1261] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a [term 1 FOLLOWER]: Refusing update from remote peer 08d2759c8a7f47e099641c26b7867c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:29:52.235157 27235 raft_consensus.cc:1261] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 1 FOLLOWER]: Refusing update from remote peer 08d2759c8a7f47e099641c26b7867c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:29:52.235852 27452 consensus_queue.cc:1035] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17dc6efc85f476fbea3761e27833bad" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:29:52.236028 27457 consensus_queue.cc:1035] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe51d2a05594c3f95f4cc823bf69a5a" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:29:52.244300 27469 mvcc.cc:205] Tried to move back new op lower bound from 7333996921788121088 to 7333996921660542976. Current Snapshot: MvccSnapshot[applied={T|T < 7333996921788121088}]
I20260927 16:29:52.244454 27468 mvcc.cc:205] Tried to move back new op lower bound from 7333996921788121088 to 7333996921660542976. Current Snapshot: MvccSnapshot[applied={T|T < 7333996921788121088}]
I20260927 16:29:52.251546 26942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:29:52.251688 26942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:29:52.253873 26942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bed5576d1efc4625878600405b5eb2dc: Sending DeleteTablet for 3 replicas of tablet 570a49ae3a3f4eac9d5a857584b392ef
I20260927 16:29:52.254666 27343 tablet_service.cc:1514] Processing DeleteTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:52 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38700
I20260927 16:29:52.254669 27227 tablet_service.cc:1514] Processing DeleteTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:52 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:41528
I20260927 16:29:52.255131 27073 tablet_service.cc:1514] Processing DeleteTablet for tablet 570a49ae3a3f4eac9d5a857584b392ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:29:52 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:57756
I20260927 16:29:52.255321 27478 tablet_replica.cc:330] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: stopping tablet replica
I20260927 16:29:52.255647 27478 raft_consensus.cc:2229] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:29:52.255698 27479 tablet_replica.cc:330] T 570a49ae3a3f4eac9d5a857584b392ef P ffe51d2a05594c3f95f4cc823bf69a5a: stopping tablet replica
I20260927 16:29:52.255900 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 27024
I20260927 16:29:52.255954 27478 raft_consensus.cc:2258] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:29:52.256239 27477 tablet_replica.cc:330] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: stopping tablet replica
I20260927 16:29:52.256680 27477 raft_consensus.cc:2229] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:29:52.257050 27477 raft_consensus.cc:2258] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:29:52.258410 27478 ts_tablet_manager.cc:1905] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:29:52.258410 27477 ts_tablet_manager.cc:1905] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:29:52.261042 27477 ts_tablet_manager.cc:1918] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:29:52.261044 27478 ts_tablet_manager.cc:1918] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:29:52.261127 27478 log.cc:1196] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/570a49ae3a3f4eac9d5a857584b392ef
I20260927 16:29:52.261133 27477 log.cc:1196] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/570a49ae3a3f4eac9d5a857584b392ef
I20260927 16:29:52.261435 27478 ts_tablet_manager.cc:1939] T 570a49ae3a3f4eac9d5a857584b392ef P c17dc6efc85f476fbea3761e27833bad: Deleting consensus metadata
I20260927 16:29:52.261435 27477 ts_tablet_manager.cc:1939] T 570a49ae3a3f4eac9d5a857584b392ef P 08d2759c8a7f47e099641c26b7867c71: Deleting consensus metadata
I20260927 16:29:52.262501 26928 catalog_manager.cc:4841] TS c17dc6efc85f476fbea3761e27833bad (127.15.93.130:32983): tablet 570a49ae3a3f4eac9d5a857584b392ef (table test-table [id=9294af76bbab4110922c3593ce1b4ed0]) successfully deleted
W20260927 16:29:52.263163 26928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:34437 (error 108)
I20260927 16:29:52.263307 26930 catalog_manager.cc:4841] TS 08d2759c8a7f47e099641c26b7867c71 (127.15.93.131:35631): tablet 570a49ae3a3f4eac9d5a857584b392ef (table test-table [id=9294af76bbab4110922c3593ce1b4ed0]) successfully deleted
I20260927 16:29:52.264110 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:29:52.263857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:34437 (local address 127.0.0.1:57758)
0927 16:29:52.263990 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 16:29:52.264042 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 16:29:52.264236 26928 catalog_manager.cc:4584] TS ffe51d2a05594c3f95f4cc823bf69a5a (127.15.93.129:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 570a49ae3a3f4eac9d5a857584b392ef: Network error: Client connection negotiation failed: client connection to 127.15.93.129:34437: connect: Connection refused (error 111)
I20260927 16:29:52.265642 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 27163
I20260927 16:29:52.271289 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 27298
I20260927 16:29:52.276755 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 26911
I20260927 16:29:52.284682 15734 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:29:52.619181 15734 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:29:52.619297 15734 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:29:52.963972 15734 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:29:52.964 UTC [26497] LOG: received smart shutdown request
2026-09-27 16:29:52.967 UTC [26497] LOG: background worker "logical replication launcher" (PID 26506) exited with exit code 1
2026-09-27 16:29:52.968 UTC [26501] LOG: shutting down
2026-09-27 16:29:53.112 UTC [26497] LOG: database system is shut down
2026-09-27T16:29:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45072 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:29:53 dist-test-slave-grzf krb5kdc[27489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:29:53 dist-test-slave-grzf krb5kdc[27489](info): set up 2 sockets
Sep 27 16:29:53 dist-test-slave-grzf krb5kdc[27489](info): commencing operation
krb5kdc: starting...
W20260927 16:29:54.267529 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:29:54 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:29:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangeradmin_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangeradmin_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangerlookup_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangerlookup_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
I20260927 16:29:54.421833 15734 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5Uwdq0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/postgres -l logfile start
2026-09-27 16:29:56.719 UTC [27519] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 16:29:56.719 UTC [27519] LOG: listening on IPv4 address "127.15.93.148", port 57103
2026-09-27 16:29:56.722 UTC [27519] LOG: listening on Unix socket "/tmp/.s.PGSQL.57103"
2026-09-27 16:29:56.743 UTC [27522] LOG: database system was shut down at 2026-09-27 16:29:55 UTC
2026-09-27 16:29:56.747 UTC [27519] LOG: database system is ready to accept connections
I20260927 16:29:57.691147 15734 mini_postgres.cc:96] Postgres bound to 57103
2026-09-27 16:29:57.697 UTC [27530] FATAL: database "slave" does not exist
127.15.93.148:57103 - accepting connections
I20260927 16:29:57.697747 15734 mini_ranger.cc:161] Starting Ranger...
I20260927 16:29:57.712077 15734 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 16:29:58.144538 15734 mini_ranger.cc:87] Created ranger Postgres database
I20260927 16:29:58.144644 15734 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin
2026-09-27 16:29:58,417 [I] DB FLAVOR :POSTGRES
2026-09-27 16:29:58,418 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:29:58,418 [I] Checking connection..
2026-09-27 16:29:58,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 16:29:58,651 [I] Checking connection passed.
2026-09-27 16:29:58,652 [I] --------- Verifying version history table ---------
2026-09-27 16:29:58,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:58,878 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:29:58,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:59,100 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 16:29:59,101 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 16:29:59,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 16:29:59.320 UTC [27594] WARNING: there is no transaction in progress
2026-09-27 16:29:59,326 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 16:29:59,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 16:29:59,561 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 16:29:59,562 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:29:59,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 16:29:59,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 16:30:00,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grzf.c.gcp-upstream.internal','N') ;"
2026-09-27 16:30:00,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:30:00,501 [I] Table x_portal_user does not exist in database ranger
2026-09-27 16:30:00,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:30:00,753 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 16:30:00,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:30:01,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:30:01,248 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 16:30:01,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 16:30:02.174 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.190 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.207 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.221 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.234 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.250 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.303 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.310 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.323 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.333 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.344 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.357 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.366 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.372 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.378 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02.384 UTC [27702] WARNING: there is no transaction in progress
2026-09-27 16:30:02,960 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 16:30:02,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 16:30:03,202 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 16:30:03,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 16:30:03,444 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 16:30:03,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 16:30:03,695 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 16:30:03,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 16:30:03,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 16:30:04,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grzf.c.gcp-upstream.internal';"
2026-09-27 16:30:04,392 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 16:30:04,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 16:30:04,610 [I] DB_PATCHES have already been applied
I20260927 16:30:04.616170 15734 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5Uwdq0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 16:30:04.616250 15734 mini_ranger.cc:193] Using host: 127.15.93.148
I20260927 16:30:04.619511 15734 mini_ranger.cc:227] Ranger admin URL: http://127.15.93.148:41703
Sep 27, 2026 4:30:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:30:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:30:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:30:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:30:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:30:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:30:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 16:30:05.730022 15734 mini_ranger.cc:160] Time spent starting Ranger: real 8.032s user 0.001s sys 0.008s
Sep 27, 2026 4:30:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:30:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 4:30:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:30:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:30:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 16:30:26.697515 15734 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.93.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.93.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangerkms_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.93.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/rangerkms_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.93.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.93.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.93.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/HTTP_127.15.93.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 16:30:26 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 16:30:26.779958 15734 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 16:30:26.788455 15734 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 16:30:27.563342 15734 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 16:30:27.563477 15734 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms
I20260927 16:30:27.563498 15734 mini_ranger_kms.cc:227] Using postgres at 127.15.93.148:57103
2026-09-27 16:30:27,732 [I] DB FLAVOR :POSTGRES
2026-09-27 16:30:27,732 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 16:30:27,732 [I] Checking connection
2026-09-27 16:30:27,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 16:30:27,988 [I] connection success
2026-09-27 16:30:27,988 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 16:30:27,988 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 16:30:27,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 16:30:28,245 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 16:30:28,245 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 16:30:28,245 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 16:30:28,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.93.148:57103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 16:30:28,490 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 16:30:28.906757 15734 mini_ranger_kms.cc:310] Created kms service
I20260927 16:30:29.087216 15734 mini_ranger_kms.cc:326] Created kudu user
I20260927 16:30:29.132154 15734 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 16:30:29 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526626, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:29.513994 15734 mini_ranger_kms.cc:384] Added ranger policy
I20260927 16:30:29.514246 15734 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms:/tmp/dist-test-task5Uwdq0/build/debug/bin/postgresql.jar:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5Uwdq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 16:30:29.514269 15734 mini_ranger_kms.cc:240] Using host: 127.15.93.148
I20260927 16:30:29.517915 15734 mini_ranger_kms.cc:276] Ranger KMS PID: 27882
I20260927 16:30:29.518009 15734 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.93.148:52169
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:30:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 16:30:31 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:31.739105 15734 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.959s user 0.023s sys 0.130s
I20260927 16:30:31.739243 15734 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 16:30:31.739317 15734 mini_ranger_kms.cc:401] 127.15.93.148:52169/kms/v1/keys
Sep 27 16:30:32 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:30:32 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526631, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.93.148@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:30:34 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:30:34.678530 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36461
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36461
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:34.789603 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:34.789868 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:34.789932 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:34.793885 27933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:34.793965 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:34.794003 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:34.794059 27933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:34.794097 27933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:34.799305 27933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.93.148:52169/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36461
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:34.800288 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:34.801478 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:34.811864 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:34.811887 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:34.812069 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:34.811990 27933 server_base.cc:1034] running on GCE node
I20260927 16:30:34.812739 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:34.813781 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:34.814950 27933 hybrid_clock.cc:648] HybridClock initialized: now 1790526634814951 us; error 45 us; skew 500 ppm
Sep 27 16:30:34 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526634, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:34.823544 27933 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:34.824680 27933 webserver.cc:460] Webserver started at http://127.15.93.190:34681/ using document root <none> and password file <none>
I20260927 16:30:34.825350 27933 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:34.825423 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:34.825640 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:30:34 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526634, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:34.879635 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0"
format_stamp: "Formatted at 2026-09-27 16:30:34 on dist-test-slave-grzf"
server_key: "aacf566ccd45773edd2dc0afba925c6f"
server_key_iv: "6afbbe4aa901f0731e1ccc30aabf9cc8"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:34.880280 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0"
format_stamp: "Formatted at 2026-09-27 16:30:34 on dist-test-slave-grzf"
server_key: "aacf566ccd45773edd2dc0afba925c6f"
server_key_iv: "6afbbe4aa901f0731e1ccc30aabf9cc8"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:34.884352 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:30:34.901505 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:34.902922 27933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:30:34.903055 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0"
format_stamp: "Formatted at 2026-09-27 16:30:34 on dist-test-slave-grzf"
server_key: "aacf566ccd45773edd2dc0afba925c6f"
server_key_iv: "6afbbe4aa901f0731e1ccc30aabf9cc8"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:34.903193 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:34.916925 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:34.919692 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:34.919873 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:34.927708 27933 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36461
I20260927 16:30:34.927712 28000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36461 every 8 connection(s)
I20260927 16:30:34.928687 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:34.931326 28001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:34.935875 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 27933
I20260927 16:30:34.936020 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:30:34.936326 15734 external_mini_cluster.cc:1428] Setting key 80e57c46e76f5d14f707ea8590b87645
I20260927 16:30:34.936865 28001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Bootstrap starting.
I20260927 16:30:34.939621 28001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:34.940562 28001 log.cc:824] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:34.942339 28001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: No bootstrap required, opened a new log
Sep 27 16:30:34 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:34.946854 28001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:34.947117 28001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:34.947175 28001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a649bb9ee8f4fae8cd7b46c1dc191c0, State: Initialized, Role: FOLLOWER
I20260927 16:30:34.947688 28001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:34.947851 28001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:34.947911 28001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:30:34.948019 28001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:34.949167 28001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:34.949527 28001 leader_election.cc:302] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a649bb9ee8f4fae8cd7b46c1dc191c0; no voters:
I20260927 16:30:34.949883 28001 leader_election.cc:288] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:30:34.949937 28006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:34.950101 28006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a649bb9ee8f4fae8cd7b46c1dc191c0, State: Running, Role: LEADER
I20260927 16:30:34.950390 28006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:34.951056 28001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:34.951504 28006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a649bb9ee8f4fae8cd7b46c1dc191c0. Latest consensus state: current_term: 1 leader_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } } }
I20260927 16:30:34.951606 28006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:34.951797 28006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } } }
I20260927 16:30:34.951897 28006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:34.952162 28010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:34.953373 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:34.937767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.15.93.190:36461)
0927 16:30:34.938216 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:34.938242 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:34.938286 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:30:34.939148 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:34.939171 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:34.939627 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:34.939981 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:34.940900 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:34.942423 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:34.943608 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:34.944303 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:34.944458 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:34.947457 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:34.947462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:34.947477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:34.947506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:34.949597 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:34.950320 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:34.950326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:34.950331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:34.950471 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:34.950862 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:34.950866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:34.950868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:34.951351 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:34.951548 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:34.952181 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 16:30:34.952453 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260927 16:30:34.955994 28010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:34.956370 28001 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:30:34.968987 28010 catalog_manager.cc:1347] Generated new cluster ID: 97792fe754604be28533f59def9fd78b
I20260927 16:30:34.969066 28010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:34.983438 28010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:30:34.984666 28010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:34.993997 28010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Generated new TSK 0
I20260927 16:30:34.994927 28010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:30:35.698796 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:35.839310 28046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:35.839668 28046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:35.839763 28046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:35.844698 28046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:35.844779 28046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:35.844862 28046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:30:35.852622 28046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:35.853943 28046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:35.855222 28046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:35.866173 28056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:35.868321 28054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:35.871467 28046 server_base.cc:1034] running on GCE node
W20260927 16:30:35.871568 28053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:35.872136 28046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:35.872946 28046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:35.874143 28046 hybrid_clock.cc:648] HybridClock initialized: now 1790526635874083 us; error 78 us; skew 500 ppm
Sep 27 16:30:35 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526635, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:35.877827 28046 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:30:35.878840 28046 webserver.cc:460] Webserver started at http://127.15.93.129:46043/ using document root <none> and password file <none>
I20260927 16:30:35.879531 28046 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:35.879627 28046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:35.879887 28046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:30:35 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526635, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:35.907428 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "5cb1f7e53f3b4177992be95408be3eda"
format_stamp: "Formatted at 2026-09-27 16:30:35 on dist-test-slave-grzf"
server_key: "43598fd14c5e2350b41423a1a69d99d3"
server_key_iv: "e6e474e8d11a75e2e6d4b2e24395ab7d"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:35.908166 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "5cb1f7e53f3b4177992be95408be3eda"
format_stamp: "Formatted at 2026-09-27 16:30:35 on dist-test-slave-grzf"
server_key: "43598fd14c5e2350b41423a1a69d99d3"
server_key_iv: "e6e474e8d11a75e2e6d4b2e24395ab7d"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:35.913067 28046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 16:30:35.943799 28065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:35.945483 28046 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260927 16:30:35.945631 28046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "5cb1f7e53f3b4177992be95408be3eda"
format_stamp: "Formatted at 2026-09-27 16:30:35 on dist-test-slave-grzf"
server_key: "43598fd14c5e2350b41423a1a69d99d3"
server_key_iv: "e6e474e8d11a75e2e6d4b2e24395ab7d"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:35.945784 28046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:35.983857 28046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 27 16:30:35 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526635, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:35.987444 28046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:35.987640 28046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:35.988291 28046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:35.989283 28046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:35.989344 28046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:35.989396 28046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:35.989424 28046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.001421 28046 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:33243
I20260927 16:30:36.002568 28046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:30:36.004402 28178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:33243 every 8 connection(s)
I20260927 16:30:36.009802 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28046
I20260927 16:30:36.009938 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:30:36.010243 15734 external_mini_cluster.cc:1428] Setting key 6973a5fb6674097a9e3e098b8cb7b3f9
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526635, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:36.022418 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.005306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:45531 (local address 127.15.93.190:36461)
0927 16:30:36.005658 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:36.005663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:36.008256 (+ 2593us) server_negotiation.cc:407] Connection header received
0927 16:30:36.008308 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:36.008311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:36.008364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:36.008445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:36.009892 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.010773 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.014556 (+ 3783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.014908 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.015031 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.018039 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:36.018043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:36.018046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:36.018073 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:36.019990 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.020615 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.020621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.020623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.020712 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.021105 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.021114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.021116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.021330 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:36.021417 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:36.022097 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 16:30:36.022230 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":92,"threads_started":1}
I20260927 16:30:36.024377 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.005757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.129:45531)
0927 16:30:36.006873 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:36.006926 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:36.007967 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:36.008742 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:36.008750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:36.009091 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:36.009729 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.009747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.010904 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.010908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.014390 (+ 3482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.014402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.015090 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.015095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.015289 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.015958 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:36.015977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:36.017847 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:36.020124 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.020130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.020147 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.020480 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.020823 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.020826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.020829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.020995 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.021486 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:36.021492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:36.021808 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 16:30:36.022872 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0927 16:30:36.023216 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":130,"threads_started":1}
I20260927 16:30:36.025779 28179 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:36.026036 28179 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:36.026592 28179 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:36.028373 27965 ts_manager.cc:194] Registered new tserver with Master: 5cb1f7e53f3b4177992be95408be3eda (127.15.93.129:33243)
I20260927 16:30:36.029850 27965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:45531
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:30:36.054224 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:36.185786 28185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:36.186148 28185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:36.186239 28185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:36.192360 28185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:36.192436 28185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:36.192531 28185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:30:36.200528 28185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:36.201957 28185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:36.203291 28185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:36.214299 28194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:36.214330 28192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:36.214512 28191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:36.214985 28185 server_base.cc:1034] running on GCE node
I20260927 16:30:36.215442 28185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:36.216189 28185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:36.217461 28185 hybrid_clock.cc:648] HybridClock initialized: now 1790526636217436 us; error 40 us; skew 500 ppm
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:36.220963 28185 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:30:36.222038 28185 webserver.cc:460] Webserver started at http://127.15.93.130:45403/ using document root <none> and password file <none>
I20260927 16:30:36.222755 28185 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:36.222851 28185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:36.223104 28185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:36.238170 28185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "7c57792526404646be7ad03597b994b1"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "22add0ab61b6a84cef4f30e452dddb18"
server_key_iv: "f385dc0b518380f700c629f1aa7a9987"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.239176 28185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "7c57792526404646be7ad03597b994b1"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "22add0ab61b6a84cef4f30e452dddb18"
server_key_iv: "f385dc0b518380f700c629f1aa7a9987"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.244510 28185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 16:30:36.267150 28201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.268940 28185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:30:36.269132 28185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "7c57792526404646be7ad03597b994b1"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "22add0ab61b6a84cef4f30e452dddb18"
server_key_iv: "f385dc0b518380f700c629f1aa7a9987"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.269338 28185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:36.311064 28185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:36.318511 28185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:36.318893 28185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:36.319639 28185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:36.320667 28185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:36.320757 28185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.320845 28185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:36.320899 28185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.344631 28185 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:44929
I20260927 16:30:36.344658 28314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:44929 every 8 connection(s)
I20260927 16:30:36.345925 28185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:30:36.350157 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28185
I20260927 16:30:36.350297 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:30:36.350621 15734 external_mini_cluster.cc:1428] Setting key 0887fa814b9c8266c5651ace78f7f132
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:36.370850 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.349666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:48133 (local address 127.15.93.190:36461)
0927 16:30:36.349796 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:36.349800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:36.356046 (+ 6246us) server_negotiation.cc:407] Connection header received
0927 16:30:36.357320 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:36.357324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:36.357379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:36.357433 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:36.359049 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.359891 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.361646 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.361998 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.362076 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.366425 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:36.366429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:36.366432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:36.366459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:36.368330 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.368949 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.368953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.368955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.369043 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.369465 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.369469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.369471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.369713 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:36.369827 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:36.370519 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 16:30:36.370677 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:30:36.373034 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.348575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.130:48133)
0927 16:30:36.355877 (+ 7302us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:36.355942 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:36.357050 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:36.357849 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:36.357859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:36.358226 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:36.358885 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.358904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.360051 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.360054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.361490 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.361501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.362297 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.362302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.362499 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.363172 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:36.363193 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:36.366227 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:36.368466 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.368472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.368489 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.368828 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.369176 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.369178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.369181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.369347 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.369833 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:36.369839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:36.370231 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 16:30:36.370448 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 16:30:36.370762 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7043,"thread_start_us":141,"threads_started":1}
I20260927 16:30:36.374537 28315 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:36.374785 28315 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:36.375355 28315 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:36.376624 27965 ts_manager.cc:194] Registered new tserver with Master: 7c57792526404646be7ad03597b994b1 (127.15.93.130:44929)
I20260927 16:30:36.377379 27965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:48133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:30:36.392107 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526635, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
W20260927 16:30:36.562726 28320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:36.563146 28320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:36.563249 28320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:36.568895 28320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:36.568989 28320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:36.569104 28320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:30:36.574565 28320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:36.575845 28320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:36.576865 28320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:36.586615 28325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:36.586699 28320 server_base.cc:1034] running on GCE node
W20260927 16:30:36.587000 28328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:36.587198 28326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:36.587661 28320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:36.588300 28320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:36.589483 28320 hybrid_clock.cc:648] HybridClock initialized: now 1790526636589452 us; error 42 us; skew 500 ppm
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:36.592450 28320 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:30:36.593497 28320 webserver.cc:460] Webserver started at http://127.15.93.131:39703/ using document root <none> and password file <none>
I20260927 16:30:36.594250 28320 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:36.594336 28320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:36.594584 28320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:36.613749 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "ca8e9fa7acf44259bc70db9276d37209"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "2927cba5b9b44a75d572ff70dc208c01"
server_key_iv: "8a71a191249bac2ae04ed22028754eab"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.614500 28320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "ca8e9fa7acf44259bc70db9276d37209"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "2927cba5b9b44a75d572ff70dc208c01"
server_key_iv: "8a71a191249bac2ae04ed22028754eab"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.620174 28320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 16:30:36.644584 28335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.646310 28320 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260927 16:30:36.646428 28320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "ca8e9fa7acf44259bc70db9276d37209"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "2927cba5b9b44a75d572ff70dc208c01"
server_key_iv: "8a71a191249bac2ae04ed22028754eab"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.646555 28320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:36.677201 28320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:36.680145 28320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:36.680361 28320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:36.681027 28320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:36.682049 28320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:36.682127 28320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.682207 28320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:36.682250 28320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.692981 28320 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:45055
I20260927 16:30:36.693194 28448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:45055 every 8 connection(s)
I20260927 16:30:36.693940 28320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:30:36.696202 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28320
I20260927 16:30:36.696326 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:30:36.696651 15734 external_mini_cluster.cc:1428] Setting key 030de18f939e605fff58d55af60aa62b
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:36.708236 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.695900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:59679 (local address 127.15.93.190:36461)
0927 16:30:36.697020 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:36.697025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:36.697623 (+ 598us) server_negotiation.cc:407] Connection header received
0927 16:30:36.698773 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:36.698777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:36.698828 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:36.698880 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:36.700171 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.700699 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.701838 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.702104 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.702211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.704526 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:36.704530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:36.704533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:36.704562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:36.705961 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.706482 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.706487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.706489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.706589 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:36.707025 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:36.707028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:36.707030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:36.707261 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:36.707364 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:36.707959 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 16:30:36.708071 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028}
I20260927 16:30:36.709117 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:36.696348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.131:59679)
0927 16:30:36.697472 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:36.697527 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:36.698541 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:36.699230 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:36.699237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:36.699562 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:36.700034 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.700047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.700823 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.700828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.701718 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:36.701726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:36.702255 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:36.702259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:36.702421 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:36.703012 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:36.703027 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:36.704372 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:36.706123 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.706128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.706143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.706378 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.706701 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:36.706704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:36.706706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:36.706910 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:36.707361 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:36.707390 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:36.707630 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 16:30:36.707869 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 16:30:36.708174 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":147,"threads_started":1}
I20260927 16:30:36.710362 28449 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:36.710608 28449 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:36.711144 28449 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:36.712182 27964 ts_manager.cc:194] Registered new tserver with Master: ca8e9fa7acf44259bc70db9276d37209 (127.15.93.131:45055)
I20260927 16:30:36.712742 27964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:59679
I20260927 16:30:36.722503 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:30:36.723474 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28046
I20260927 16:30:36.732762 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28185
I20260927 16:30:36.739184 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28320
I20260927 16:30:36.745173 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 27933
I20260927 16:30:36.750692 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36461
--webserver_interface=127.15.93.190
--webserver_port=34681
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36461
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:36.857889 28453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:36.858162 28453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:36.858201 28453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:36.862149 28453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:36.862202 28453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:36.862221 28453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:36.862237 28453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:36.862252 28453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:36.867575 28453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36461
--ranger_config_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:36.868490 28453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:36.869402 28453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:36.877434 28459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:36.877446 28461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:36.877550 28458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:36.877620 28453 server_base.cc:1034] running on GCE node
I20260927 16:30:36.878232 28453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:36.879230 28453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:36.880417 28453 hybrid_clock.cc:648] HybridClock initialized: now 1790526636880409 us; error 34 us; skew 500 ppm
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:36.883006 28453 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:36.883939 28453 webserver.cc:460] Webserver started at http://127.15.93.190:34681/ using document root <none> and password file <none>
I20260927 16:30:36.884459 28453 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:36.884552 28453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:30:36 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:36.943823 28453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:30:36.946023 28469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:36.947307 28453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:36.947468 28453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0"
format_stamp: "Formatted at 2026-09-27 16:30:34 on dist-test-slave-grzf"
server_key: "aacf566ccd45773edd2dc0afba925c6f"
server_key_iv: "6afbbe4aa901f0731e1ccc30aabf9cc8"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:36.948015 28453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:36.956602 28453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:36.958942 28453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:36.959125 28453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:36.965955 28453 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36461
I20260927 16:30:36.965960 28521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36461 every 8 connection(s)
I20260927 16:30:36.966984 28453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:36.967175 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28453
I20260927 16:30:36.967706 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:33243
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:36461
--webserver_port=46043
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:36.971094 28522 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:30:36.972748 28522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Bootstrap starting.
I20260927 16:30:36.988508 28522 log.cc:824] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:36.994719 28522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:36.995110 28522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Bootstrap complete.
I20260927 16:30:36.999060 28522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:36.999501 28522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a649bb9ee8f4fae8cd7b46c1dc191c0, State: Initialized, Role: FOLLOWER
I20260927 16:30:36.999953 28522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:37.000092 28522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:37.000183 28522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:30:37.000293 28522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:30:37.002048 28522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:37.002370 28522 leader_election.cc:302] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a649bb9ee8f4fae8cd7b46c1dc191c0; no voters:
I20260927 16:30:37.002751 28522 leader_election.cc:288] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:30:37.002831 28526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:30:37.003019 28526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [term 2 LEADER]: Becoming Leader. State: Replica: 0a649bb9ee8f4fae8cd7b46c1dc191c0, State: Running, Role: LEADER
I20260927 16:30:37.003293 28526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } }
I20260927 16:30:37.003930 28522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:37.004184 28526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a649bb9ee8f4fae8cd7b46c1dc191c0. Latest consensus state: current_term: 2 leader_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } } }
I20260927 16:30:37.004297 28526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:37.004686 28527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a649bb9ee8f4fae8cd7b46c1dc191c0" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36461 } } }
I20260927 16:30:37.004844 28527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:37.005882 28530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:37.006204 28522 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 16:30:37.012290 28530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:37.012977 28530 catalog_manager.cc:1259] Loaded cluster ID: 97792fe754604be28533f59def9fd78b
I20260927 16:30:37.013067 28530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:37.015543 28530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:37.016239 28530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Loaded TSK: 0
I20260927 16:30:37.017467 28530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:30:37.083214 28524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:37.083532 28524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:37.083616 28524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:37.090540 28524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:37.090637 28524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:37.090730 28524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:30:37.099098 28524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:33243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=46043
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:37.100463 28524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:37.101749 28524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:37.114241 28560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.114311 28559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.114673 28562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:37.115330 28524 server_base.cc:1034] running on GCE node
I20260927 16:30:37.115795 28524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:37.116457 28524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:37.119728 28524 hybrid_clock.cc:648] HybridClock initialized: now 1790526637119692 us; error 45 us; skew 500 ppm
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:37.122731 28524 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:30:37.123798 28524 webserver.cc:460] Webserver started at http://127.15.93.129:46043/ using document root <none> and password file <none>
I20260927 16:30:37.124365 28524 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:37.124469 28524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:37.147019 28524 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 16:30:37.153667 28570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.155493 28524 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s
I20260927 16:30:37.155661 28524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "5cb1f7e53f3b4177992be95408be3eda"
format_stamp: "Formatted at 2026-09-27 16:30:35 on dist-test-slave-grzf"
server_key: "43598fd14c5e2350b41423a1a69d99d3"
server_key_iv: "e6e474e8d11a75e2e6d4b2e24395ab7d"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:37.156437 28524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:37.177974 28524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:37.181614 28524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:37.181820 28524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:37.182516 28524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:37.183565 28524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:37.183629 28524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.183686 28524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:37.183717 28524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.193706 28524 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:33243
I20260927 16:30:37.194911 28524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:30:37.196120 28683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:33243 every 8 connection(s)
I20260927 16:30:37.204385 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28524
I20260927 16:30:37.204941 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:44929
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:36461
--webserver_port=45403
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:37.223453 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.198221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.129:34513)
0927 16:30:37.204884 (+ 6663us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:37.204950 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:37.205986 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:37.207997 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:37.208005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:37.208335 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:37.208902 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.208917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.210755 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.210758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.212205 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.212216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.213396 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.213400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.213575 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.214288 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:37.214308 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:37.218028 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:37.220275 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.220281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.220295 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.220551 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.220901 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.220906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.220908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.221067 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.221679 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:37.221685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:37.221920 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 16:30:37.222100 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 16:30:37.222429 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6411,"thread_start_us":140,"threads_started":1}
I20260927 16:30:37.223445 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.198166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:34513 (local address 127.15.93.190:36461)
0927 16:30:37.206484 (+ 8318us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:37.206505 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:37.206545 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:30:37.206824 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:37.206837 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:37.207220 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:37.207554 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:37.209103 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.210583 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.212337 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.213007 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.213156 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.218224 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:37.218230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:37.218252 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:37.218279 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:37.220159 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.220679 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.220691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.220695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.220805 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.221194 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.221198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.221199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.221554 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:37.221694 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:37.222227 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 16:30:37.222460 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8153,"thread_start_us":114,"threads_started":1}
I20260927 16:30:37.226271 28684 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:37.226536 28684 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.227072 28684 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.229228 28684 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:36461 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:30:37.229341 28684 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.229494 28684 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:37.230608 28684 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.230755 28684 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.231597 28684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:30:37.365149 28688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:37.365418 28688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:37.365485 28688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:37.369660 28688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:37.369737 28688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:37.369827 28688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:30:37.375291 28688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:44929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=45403
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:37.376494 28688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:37.377596 28688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:37.388106 28696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.388149 28694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.388417 28693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:37.388458 28688 server_base.cc:1034] running on GCE node
I20260927 16:30:37.388835 28688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:37.389436 28688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:37.390623 28688 hybrid_clock.cc:648] HybridClock initialized: now 1790526637390596 us; error 46 us; skew 500 ppm
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:37.393859 28688 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:30:37.394907 28688 webserver.cc:460] Webserver started at http://127.15.93.130:45403/ using document root <none> and password file <none>
I20260927 16:30:37.395618 28688 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:37.395723 28688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:37.416394 28688 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 16:30:37.422366 28704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.424054 28688 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 16:30:37.424191 28688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "7c57792526404646be7ad03597b994b1"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "22add0ab61b6a84cef4f30e452dddb18"
server_key_iv: "f385dc0b518380f700c629f1aa7a9987"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:37.424908 28688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:37.451784 28688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:37.455339 28688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:37.455557 28688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:37.456245 28688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:37.457123 28688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:37.457171 28688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.457209 28688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:37.457231 28688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.468477 28688 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:44929
I20260927 16:30:37.469698 28688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:30:37.471498 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:44929 every 8 connection(s)
I20260927 16:30:37.475128 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28688
I20260927 16:30:37.475728 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.93.148:52169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:45055
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:36461
--webserver_port=39703
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:37.486958 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.473347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:33907 (local address 127.15.93.190:36461)
0927 16:30:37.473517 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:37.473521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:37.473541 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:30:37.474111 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:37.474114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:37.474167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:37.474261 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:37.475661 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.476498 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.478148 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.478504 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.478618 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.481658 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:37.481662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:37.481665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:37.481693 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:37.483457 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.484047 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.484051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.484053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.484149 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.484515 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.484518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.484520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.484727 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:37.484814 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:37.486693 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0927 16:30:37.486815 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 16:30:37.487167 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.472217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.130:33907)
0927 16:30:37.472780 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:37.472834 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:37.473853 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:37.474471 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:37.474480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:37.474831 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:37.475485 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.475505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.476655 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.476658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.478009 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.478019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.478781 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.478785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.478982 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.479750 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:37.479770 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:37.481468 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:37.483586 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.483592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.483607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.483931 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.484249 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.484252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.484254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.484414 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.484863 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:37.484870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:37.485158 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 16:30:37.485346 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 16:30:37.485643 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":144,"threads_started":1}
I20260927 16:30:37.488555 28818 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:37.488807 28818 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.489389 28818 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.490902 28818 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:36461 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:30:37.491055 28818 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.491235 28818 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:37.492086 28818 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.492291 28818 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.493040 28818 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:30:37.640828 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:37.641196 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:37.641266 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:37.647449 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:37.647532 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:37.647624 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:30:37.655759 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.93.148:52169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=39703
--tserver_master_addrs=127.15.93.190:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:37.657109 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:37.658437 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:37.672554 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.675959 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:37.676013 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:37.677209 28821 server_base.cc:1034] running on GCE node
I20260927 16:30:37.677708 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:37.678509 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:37.679703 28821 hybrid_clock.cc:648] HybridClock initialized: now 1790526637679672 us; error 48 us; skew 500 ppm
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:37.683274 28821 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:30:37.684218 28821 webserver.cc:460] Webserver started at http://127.15.93.131:39703/ using document root <none> and password file <none>
I20260927 16:30:37.684831 28821 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:37.684979 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:37.708858 28821 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260927 16:30:37.712065 28838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.713449 28821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:30:37.713588 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "ca8e9fa7acf44259bc70db9276d37209"
format_stamp: "Formatted at 2026-09-27 16:30:36 on dist-test-slave-grzf"
server_key: "2927cba5b9b44a75d572ff70dc208c01"
server_key_iv: "8a71a191249bac2ae04ed22028754eab"
server_key_version: "kuduclusterkey@0"
I20260927 16:30:37.714165 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:37.729331 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:37.732897 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:37.733155 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:37.733861 28821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:37.734956 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:37.735059 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.735168 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:37.735235 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:37.746619 28821 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:45055
I20260927 16:30:37.746688 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:45055 every 8 connection(s)
I20260927 16:30:37.747799 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:30:37.756270 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 28821
Sep 27 16:30:37 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:37.764578 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.751582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:41977 (local address 127.15.93.190:36461)
0927 16:30:37.751747 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:37.751752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:37.751774 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:30:37.752997 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:37.753001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:37.753057 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:37.753131 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:37.754747 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.755598 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.757447 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.757794 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.757869 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.760781 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:37.760785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:37.760788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:37.760814 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:37.762479 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.762984 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.762988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.762989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.763070 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.763423 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.763427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.763432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.763626 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:37.763732 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:37.764254 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 16:30:37.764377 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 16:30:37.765848 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.750345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36461 (local address 127.15.93.131:41977)
0927 16:30:37.751636 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:37.751693 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:37.752748 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:37.753536 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:37.753545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:37.753892 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:37.754571 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.754592 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.755754 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.755757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.757296 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:37.757307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.758059 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.758063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:37.758276 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.758940 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:37.758960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:37.760620 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:37.762582 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.762590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.762606 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.762896 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.763168 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:37.763171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:37.763173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:37.763312 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:37.763731 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:37.763737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:37.763976 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 16:30:37.764798 (+ 822us) client_negotiation.cc:240] Negotiation successful
0927 16:30:37.765132 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"thread_start_us":124,"threads_started":1}
I20260927 16:30:37.767071 28952 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36461
I20260927 16:30:37.767328 28952 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.767948 28952 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.769632 28952 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:36461 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 16:30:37.769761 28952 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.769964 28952 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:37.770942 28952 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:37.771128 28952 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
W20260927 16:30:37.771908 28952 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:30:37.776507 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:37.756936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.15.93.190:36461)
0927 16:30:37.763531 (+ 6595us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:37.763535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:37.763810 (+ 275us) server_negotiation.cc:407] Connection header received
0927 16:30:37.763973 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:37.763979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:37.764029 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:37.764114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:37.765039 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.765843 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.767474 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:37.767695 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:37.767812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:37.769294 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:37.769306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:37.769312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:37.769337 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:37.771260 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.775183 (+ 3923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.775187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.775189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.775277 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:37.775634 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:37.775637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:37.775639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:37.775863 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:37.776066 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:37.776193 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 16:30:37.776323 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6514,"thread_start_us":91,"threads_started":1}
Sep 27 16:30:38 dist-test-slave-grzf krb5kdc[27489](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790526638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:38.231819 28684 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:38.232141 28684 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:38.233408 28486 ts_manager.cc:194] Registered new tserver with Master: 5cb1f7e53f3b4177992be95408be3eda (127.15.93.129:33243)
I20260927 16:30:38.235040 28486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:34513
I20260927 16:30:38.493278 28818 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:38.493577 28818 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:38.494328 28486 ts_manager.cc:194] Registered new tserver with Master: 7c57792526404646be7ad03597b994b1 (127.15.93.130:44929)
I20260927 16:30:38.495150 28486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:33907
Sep 27 16:30:38 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
Sep 27 16:30:38 dist-test-slave-grzf krb5kdc[27489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for HTTP/127.15.93.148@KRBTEST.COM
I20260927 16:30:38.774127 28952 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:38.774410 28952 heartbeater.cc:511] Master 127.15.93.190:36461 requested a full tablet report, sending...
I20260927 16:30:38.775204 28486 ts_manager.cc:194] Registered new tserver with Master: ca8e9fa7acf44259bc70db9276d37209 (127.15.93.131:45055)
I20260927 16:30:38.776021 28486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:41977
I20260927 16:30:38.779908 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:30:38.799242 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.783902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.15.93.190:36461)
0927 16:30:38.784232 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:38.784237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:38.787438 (+ 3201us) server_negotiation.cc:407] Connection header received
0927 16:30:38.787582 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:38.787585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:38.787642 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:38.787729 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:38.788577 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.789354 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.790381 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.791163 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.791258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.793018 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:38.793022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:38.793025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:38.793052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:38.796262 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:38.797007 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:38.797011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:38.797013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:38.797396 (+ 383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:38.797788 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:38.797792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:38.797797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:38.798027 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:38.798160 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:38.798931 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 16:30:38.799062 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":94,"threads_started":1}
I20260927 16:30:38.803411 28486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:30:38.808578 28486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:30:38.843040 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.834172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:44929 (local address 127.0.0.1:48984)
0927 16:30:38.837834 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:38.837873 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:38.838036 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:38.838533 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:38.838540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:38.838562 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:38.838701 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.838708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.839820 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.839823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.841162 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.841173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.842459 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.842462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.842574 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.842643 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 16:30:38.842745 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 16:30:38.842868 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3506,"thread_start_us":99,"threads_started":1}
I20260927 16:30:38.843724 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.835583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.15.93.130:44929)
0927 16:30:38.835920 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:38.835927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:38.838156 (+ 2229us) server_negotiation.cc:407] Connection header received
0927 16:30:38.838224 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:38.838227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:38.838390 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:38.838542 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:38.838829 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.839702 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.841307 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.842136 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.842239 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.842392 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:38.843478 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0927 16:30:38.843581 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260927 16:30:38.847344 28745 tablet_service.cc:1467] Processing CreateTablet for tablet 1881074e067c4e8da417778f39d2962d (DEFAULT_TABLE table=test-table [id=d14edd22f7614cbd8a736c4e5afa8447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:38.848512 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881074e067c4e8da417778f39d2962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:38.860013 28988 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Bootstrap starting.
I20260927 16:30:38.863024 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.840927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.15.93.131:45055)
0927 16:30:38.852529 (+ 11602us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:38.852536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:38.856749 (+ 4213us) server_negotiation.cc:407] Connection header received
0927 16:30:38.857009 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:38.857016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:38.857235 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:38.857367 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:38.857806 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.858911 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.861123 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.862073 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.862155 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.862298 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:38.862756 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 16:30:38.862862 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11476,"thread_start_us":105,"threads_started":1}
I20260927 16:30:38.863713 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.834202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:45055 (local address 127.0.0.1:56744)
0927 16:30:38.856637 (+ 22435us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:38.856654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:38.856840 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:38.857553 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:38.857556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:38.857573 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:38.857677 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.857684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.859039 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.859042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.860985 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.860996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.862394 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.862398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.862492 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.862517 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:30:38.863494 (+ 977us) client_negotiation.cc:240] Negotiation successful
0927 16:30:38.863573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22357,"thread_start_us":90,"threads_started":1}
I20260927 16:30:38.864212 28988 tablet_bootstrap.cc:654] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:38.865681 28988 log.cc:824] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:38.868426 28988 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: No bootstrap required, opened a new log
I20260927 16:30:38.868547 28886 tablet_service.cc:1467] Processing CreateTablet for tablet 1881074e067c4e8da417778f39d2962d (DEFAULT_TABLE table=test-table [id=d14edd22f7614cbd8a736c4e5afa8447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:38.869673 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881074e067c4e8da417778f39d2962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:38.871016 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.835121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.15.93.129:33243)
0927 16:30:38.835455 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:38.835463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:38.856940 (+ 21477us) server_negotiation.cc:407] Connection header received
0927 16:30:38.857053 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:38.857057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:38.857235 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:38.857374 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:38.858809 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.860013 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.864431 (+ 4418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.865138 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.865207 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.865304 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:38.870748 (+ 5444us) server_negotiation.cc:299] Negotiation successful
0927 16:30:38.870874 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":141,"threads_started":1}
I20260927 16:30:38.875408 28988 ts_tablet_manager.cc:1397] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260927 16:30:38.879700 28988 raft_consensus.cc:348] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.880018 28988 raft_consensus.cc:374] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:38.880087 28988 raft_consensus.cc:729] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c57792526404646be7ad03597b994b1, State: Initialized, Role: FOLLOWER
I20260927 16:30:38.880563 28988 consensus_queue.cc:260] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.880651 28990 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Bootstrap starting.
I20260927 16:30:38.881677 28988 ts_tablet_manager.cc:1428] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:30:38.882056 28818 heartbeater.cc:503] Master 127.15.93.190:36461 was elected leader, sending a full tablet report...
I20260927 16:30:38.883474 28990 tablet_bootstrap.cc:654] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:38.884383 28990 log.cc:824] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:38.889133 28990 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: No bootstrap required, opened a new log
I20260927 16:30:38.889377 28990 ts_tablet_manager.cc:1397] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260927 16:30:38.889457 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:38.834964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:33243 (local address 127.0.0.1:59464)
0927 16:30:38.856822 (+ 21858us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:38.856837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:38.856950 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:38.858075 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:38.858078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:38.858109 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:38.858327 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.858334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.861618 (+ 3284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.861621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.863856 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:38.864313 (+ 457us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:38.870423 (+ 6110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:38.870427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:38.870533 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:38.870556 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:30:38.871253 (+ 697us) client_negotiation.cc:240] Negotiation successful
0927 16:30:38.886296 (+ 15043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21784,"thread_start_us":99,"threads_started":1}
I20260927 16:30:38.892784 28600 tablet_service.cc:1467] Processing CreateTablet for tablet 1881074e067c4e8da417778f39d2962d (DEFAULT_TABLE table=test-table [id=d14edd22f7614cbd8a736c4e5afa8447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:38.893415 28990 raft_consensus.cc:348] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.893697 28990 raft_consensus.cc:374] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:38.893745 28990 raft_consensus.cc:729] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8e9fa7acf44259bc70db9276d37209, State: Initialized, Role: FOLLOWER
I20260927 16:30:38.893929 28600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881074e067c4e8da417778f39d2962d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:38.894209 28990 consensus_queue.cc:260] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.895193 28990 ts_tablet_manager.cc:1428] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260927 16:30:38.895531 28952 heartbeater.cc:503] Master 127.15.93.190:36461 was elected leader, sending a full tablet report...
I20260927 16:30:38.903674 28994 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Bootstrap starting.
I20260927 16:30:38.906301 28994 tablet_bootstrap.cc:654] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:38.907171 28994 log.cc:824] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:38.909818 28994 tablet_bootstrap.cc:492] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: No bootstrap required, opened a new log
I20260927 16:30:38.910014 28994 ts_tablet_manager.cc:1397] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 16:30:38.913990 28994 raft_consensus.cc:348] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.914254 28994 raft_consensus.cc:374] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:38.914294 28994 raft_consensus.cc:729] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb1f7e53f3b4177992be95408be3eda, State: Initialized, Role: FOLLOWER
I20260927 16:30:38.914780 28994 consensus_queue.cc:260] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:38.915990 28994 ts_tablet_manager.cc:1428] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:30:38.916397 28684 heartbeater.cc:503] Master 127.15.93.190:36461 was elected leader, sending a full tablet report...
W20260927 16:30:38.950798 28685 tablet.cc:2369] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:30:38.972565 28819 tablet.cc:2369] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:30:38.999863 28953 tablet.cc:2369] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:30:39.255440 28991 raft_consensus.cc:482] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:39.255687 28991 raft_consensus.cc:504] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:39.257052 28991 leader_election.cc:288] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8e9fa7acf44259bc70db9276d37209 (127.15.93.131:45055), 5cb1f7e53f3b4177992be95408be3eda (127.15.93.129:33243)
I20260927 16:30:39.260959 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.257751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:34923 (local address 127.15.93.129:33243)
0927 16:30:39.257872 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:39.257875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:39.257885 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:30:39.257954 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:39.257957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:39.258001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:39.258079 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:39.258386 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.258858 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.259974 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.260374 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.260495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:39.260546 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:39.260757 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 16:30:39.260826 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:30:39.260983 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.257255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:45055 (local address 127.15.93.130:53511)
0927 16:30:39.257549 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:39.257565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:39.257692 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:39.257952 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:39.257955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:39.257972 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:39.258146 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:39.258155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.258933 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.258938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:39.259869 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:39.259878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.260518 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.260521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:39.260654 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:39.260672 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:30:39.260758 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 16:30:39.260825 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":38,"thread_start_us":131,"threads_started":1}
I20260927 16:30:39.261013 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.257393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:53511 (local address 127.15.93.131:45055)
0927 16:30:39.257523 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:39.257529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:39.257656 (+ 127us) server_negotiation.cc:407] Connection header received
0927 16:30:39.257790 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:39.257793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:39.257850 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:39.257922 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:39.258274 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.258826 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.260014 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.260389 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.260515 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:39.260568 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:39.260828 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 16:30:39.260904 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:30:39.260990 29005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.257332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:33243 (local address 127.15.93.130:34923)
0927 16:30:39.257678 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:39.257693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:39.257787 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:39.258110 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:39.258113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:39.258126 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:39.258273 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:39.258280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.258997 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.258999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:39.259857 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:39.259866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.260490 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.260494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:39.260598 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:39.260660 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 16:30:39.260718 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 16:30:39.260812 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":65,"thread_start_us":105,"threads_started":1}
I20260927 16:30:39.261822 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881074e067c4e8da417778f39d2962d" candidate_uuid: "7c57792526404646be7ad03597b994b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb1f7e53f3b4177992be95408be3eda" is_pre_election: true
I20260927 16:30:39.261891 28888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881074e067c4e8da417778f39d2962d" candidate_uuid: "7c57792526404646be7ad03597b994b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8e9fa7acf44259bc70db9276d37209" is_pre_election: true
I20260927 16:30:39.262140 28620 raft_consensus.cc:2454] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c57792526404646be7ad03597b994b1 in term 0.
I20260927 16:30:39.262190 28888 raft_consensus.cc:2454] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c57792526404646be7ad03597b994b1 in term 0.
I20260927 16:30:39.262616 28706 leader_election.cc:302] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb1f7e53f3b4177992be95408be3eda, 7c57792526404646be7ad03597b994b1, ca8e9fa7acf44259bc70db9276d37209; no voters:
I20260927 16:30:39.262883 28991 raft_consensus.cc:2781] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:30:39.262962 28991 raft_consensus.cc:482] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:30:39.263013 28991 raft_consensus.cc:3037] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:39.264089 28991 raft_consensus.cc:504] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:39.264422 28991 leader_election.cc:288] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [CANDIDATE]: Term 1 election: Requested vote from peers ca8e9fa7acf44259bc70db9276d37209 (127.15.93.131:45055), 5cb1f7e53f3b4177992be95408be3eda (127.15.93.129:33243)
I20260927 16:30:39.264753 28620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881074e067c4e8da417778f39d2962d" candidate_uuid: "7c57792526404646be7ad03597b994b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb1f7e53f3b4177992be95408be3eda"
I20260927 16:30:39.264753 28888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881074e067c4e8da417778f39d2962d" candidate_uuid: "7c57792526404646be7ad03597b994b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8e9fa7acf44259bc70db9276d37209"
I20260927 16:30:39.264870 28620 raft_consensus.cc:3037] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:39.264870 28888 raft_consensus.cc:3037] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:39.266060 28888 raft_consensus.cc:2454] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c57792526404646be7ad03597b994b1 in term 1.
I20260927 16:30:39.266060 28620 raft_consensus.cc:2454] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c57792526404646be7ad03597b994b1 in term 1.
I20260927 16:30:39.266317 28706 leader_election.cc:302] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb1f7e53f3b4177992be95408be3eda, 7c57792526404646be7ad03597b994b1; no voters:
I20260927 16:30:39.266460 28991 raft_consensus.cc:2781] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:39.266651 28991 raft_consensus.cc:686] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 1 LEADER]: Becoming Leader. State: Replica: 7c57792526404646be7ad03597b994b1, State: Running, Role: LEADER
I20260927 16:30:39.266989 28991 consensus_queue.cc:237] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } }
I20260927 16:30:39.270148 28486 catalog_manager.cc:5495] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c57792526404646be7ad03597b994b1 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "7c57792526404646be7ad03597b994b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c57792526404646be7ad03597b994b1" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 } health_report { overall_health: UNKNOWN } } }
I20260927 16:30:39.294270 28986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.290667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.15.93.130:44929)
0927 16:30:39.290810 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:39.290816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:39.290830 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:30:39.291015 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:39.291018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:39.291079 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:39.291192 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:30:39.291469 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.291990 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.292835 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:39.293066 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:39.293190 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:39.293479 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:30:39.293866 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:30:39.293992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:39.294054 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 16:30:39.294113 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:30:39.299708 28888 raft_consensus.cc:1261] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 1 FOLLOWER]: Refusing update from remote peer 7c57792526404646be7ad03597b994b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:39.299708 28620 raft_consensus.cc:1261] T 1881074e067c4e8da417778f39d2962d P 5cb1f7e53f3b4177992be95408be3eda [term 1 FOLLOWER]: Refusing update from remote peer 7c57792526404646be7ad03597b994b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:39.300329 29006 consensus_queue.cc:1035] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb1f7e53f3b4177992be95408be3eda" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:39.300511 28991 consensus_queue.cc:1035] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8e9fa7acf44259bc70db9276d37209" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:39.307542 29009 mvcc.cc:205] Tried to move back new op lower bound from 7333997114564837376 to 7333997114440380416. Current Snapshot: MvccSnapshot[applied={T|T < 7333997114564837376}]
I20260927 16:30:39.315697 28485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:39.315841 28485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:39.318084 28485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a649bb9ee8f4fae8cd7b46c1dc191c0: Sending DeleteTablet for 3 replicas of tablet 1881074e067c4e8da417778f39d2962d
I20260927 16:30:39.318897 28886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1881074e067c4e8da417778f39d2962d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:39 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:56744
I20260927 16:30:39.318892 28600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1881074e067c4e8da417778f39d2962d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:39 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:59464
I20260927 16:30:39.319510 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28524
I20260927 16:30:39.319562 28745 tablet_service.cc:1514] Processing DeleteTablet for tablet 1881074e067c4e8da417778f39d2962d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:39 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48984
I20260927 16:30:39.320168 29019 tablet_replica.cc:330] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: stopping tablet replica
I20260927 16:30:39.320495 29019 raft_consensus.cc:2229] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:39.320735 29021 tablet_replica.cc:330] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: stopping tablet replica
I20260927 16:30:39.320830 29019 raft_consensus.cc:2258] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:39.321125 29021 raft_consensus.cc:2229] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:30:39.321523 29021 raft_consensus.cc:2258] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:39.322443 29019 ts_tablet_manager.cc:1905] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:39.323021 29021 ts_tablet_manager.cc:1905] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:39.325232 29019 ts_tablet_manager.cc:1918] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:39.325333 29019 log.cc:1196] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/1881074e067c4e8da417778f39d2962d
I20260927 16:30:39.325681 29019 ts_tablet_manager.cc:1939] T 1881074e067c4e8da417778f39d2962d P ca8e9fa7acf44259bc70db9276d37209: Deleting consensus metadata
I20260927 16:30:39.325953 29021 ts_tablet_manager.cc:1918] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:39.326110 29021 log.cc:1196] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/1881074e067c4e8da417778f39d2962d
I20260927 16:30:39.326464 29021 ts_tablet_manager.cc:1939] T 1881074e067c4e8da417778f39d2962d P 7c57792526404646be7ad03597b994b1: Deleting consensus metadata
W20260927 16:30:39.326941 28471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:33243 (error 108)
I20260927 16:30:39.326969 28470 catalog_manager.cc:4841] TS ca8e9fa7acf44259bc70db9276d37209 (127.15.93.131:45055): tablet 1881074e067c4e8da417778f39d2962d (table test-table [id=d14edd22f7614cbd8a736c4e5afa8447]) successfully deleted
I20260927 16:30:39.327646 28471 catalog_manager.cc:4841] TS 7c57792526404646be7ad03597b994b1 (127.15.93.130:44929): tablet 1881074e067c4e8da417778f39d2962d (table test-table [id=d14edd22f7614cbd8a736c4e5afa8447]) successfully deleted
I20260927 16:30:39.327860 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28688
I20260927 16:30:39.329967 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:39.329633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:33243 (local address 127.0.0.1:59478)
0927 16:30:39.329788 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:39.329869 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 16:30:39.330135 28471 catalog_manager.cc:4584] TS 5cb1f7e53f3b4177992be95408be3eda (127.15.93.129:33243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1881074e067c4e8da417778f39d2962d: Network error: Client connection negotiation failed: client connection to 127.15.93.129:33243: connect: Connection refused (error 111)
I20260927 16:30:39.333653 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28821
I20260927 16:30:39.338940 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 28453
I20260927 16:30:39.345490 15734 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 16:30:39.690024 15734 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 16:30:39.690192 15734 mini_ranger.cc:66] Stopping Ranger...
I20260927 16:30:40.035665 15734 mini_ranger.cc:68] Stopped Ranger
2026-09-27 16:30:40.035 UTC [27519] LOG: received smart shutdown request
2026-09-27 16:30:40.039 UTC [27519] LOG: background worker "logical replication launcher" (PID 27528) exited with exit code 1
2026-09-27 16:30:40.040 UTC [27523] LOG: shutting down
2026-09-27 16:30:40.184 UTC [27519] LOG: database system is shut down
2026-09-27T16:30:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47072 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:30:40 dist-test-slave-grzf krb5kdc[29031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:30:40 dist-test-slave-grzf krb5kdc[29031](info): set up 2 sockets
Sep 27 16:30:40 dist-test-slave-grzf krb5kdc[29031](info): commencing operation
krb5kdc: starting...
W20260927 16:30:41.331321 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:30:41.451292 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34499
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:41.557760 29044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:41.558051 29044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:41.558152 29044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:41.562160 29044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:41.562244 29044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:41.562282 29044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:41.562341 29044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:41.562381 29044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:41.567615 29044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:41.568627 29044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:41.569620 29044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:41.577687 29050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:41.577693 29049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.577903 29044 server_base.cc:1034] running on GCE node
W20260927 16:30:41.577759 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.578519 29044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:41.579589 29044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:41.580749 29044 hybrid_clock.cc:648] HybridClock initialized: now 1790526641580732 us; error 29 us; skew 500 ppm
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:41.583283 29044 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:41.584115 29044 webserver.cc:460] Webserver started at http://127.15.93.190:44671/ using document root <none> and password file <none>
I20260927 16:30:41.584651 29044 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:41.584745 29044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:41.584986 29044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:41.586913 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "afb9240c4887402bb1339d8153fa63d4"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dff9c731f48c35a25c4a3a5a6b064ae8"
server_key_iv: "2c35f1281109c20cf160baeab16e5042"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.587518 29044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "afb9240c4887402bb1339d8153fa63d4"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dff9c731f48c35a25c4a3a5a6b064ae8"
server_key_iv: "2c35f1281109c20cf160baeab16e5042"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.590888 29044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:30:41.593123 29059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:41.594266 29044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:41.594405 29044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "afb9240c4887402bb1339d8153fa63d4"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dff9c731f48c35a25c4a3a5a6b064ae8"
server_key_iv: "2c35f1281109c20cf160baeab16e5042"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.594521 29044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:41.615325 29044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:41.618070 29044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:41.618278 29044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:41.625344 29044 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:34499
I20260927 16:30:41.625360 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:34499 every 8 connection(s)
I20260927 16:30:41.626363 29044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:41.626834 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29044
I20260927 16:30:41.626986 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:30:41.627280 15734 external_mini_cluster.cc:1428] Setting key f5d3ed1bdea61f8876601070412c60c2
I20260927 16:30:41.629172 29112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:41.632905 29112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.93.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.189@KRBTEST.COM" created.
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:41.648304 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.636191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.15.93.190:34499)
0927 16:30:41.636522 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:41.636534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:41.636579 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:30:41.636929 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:41.636933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:41.637107 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:41.637293 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:41.638146 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.639164 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.640298 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.640681 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.640847 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.643458 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:41.643464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:41.643475 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:41.643495 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:41.645095 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.645578 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.645584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.645586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.645690 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.645997 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.646002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.646004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.646366 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:41.646506 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:41.646861 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 16:30:41.647242 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
I20260927 16:30:41.648304 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.635231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34499 (local address 127.0.0.1:47052)
0927 16:30:41.635725 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.635766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:41.636635 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:41.637287 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:41.637292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:41.637750 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:41.637984 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.637997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.639333 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.639340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.640167 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.640177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.640830 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.640834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.640961 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.641578 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:41.641605 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:41.643284 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:41.645221 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.645231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.645241 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.645461 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.645788 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.645790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.645792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.645905 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.646486 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:41.646492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:41.646631 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 16:30:41.646802 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 16:30:41.647196 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":81,"spinlock_wait_cycles":1920,"thread_start_us":154,"threads_started":1}
I20260927 16:30:41.650938 29112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:30:41.651692 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.651359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54378)
0927 16:30:41.651549 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.651628 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 16:30:41.651890 29061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
I20260927 16:30:41.652673 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.652470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54384)
0927 16:30:41.652568 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.652613 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:30:41.652834 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:37707: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
I20260927 16:30:41.654241 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:37707
--webserver_interface=127.15.93.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:41.702119 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } attempt: 1
I20260927 16:30:41.703055 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.702723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54390)
0927 16:30:41.702899 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.702965 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 16:30:41.703858 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.703659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54392)
0927 16:30:41.703759 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.703803 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 16:30:41.704003 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:37707: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
W20260927 16:30:41.763103 29119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:41.763356 29119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:41.763464 29119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:41.767408 29119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:41.767469 29119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:41.767488 29119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:41.767504 29119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:41.767519 29119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:41.772208 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } attempt: 2
I20260927 16:30:41.773096 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.772778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54408)
0927 16:30:41.772948 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.773021 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 16:30:41.773121 29119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.93.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:41.773974 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.773647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54416)
0927 16:30:41.773851 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.773913 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260927 16:30:41.774187 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:37707: Network error: Client connection negotiation failed: client connection to 127.15.93.189:37707: connect: Connection refused (error 111)
I20260927 16:30:41.774497 29119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:41.775518 29119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:41.783638 29124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:41.783624 29125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.783680 29119 server_base.cc:1034] running on GCE node
W20260927 16:30:41.783624 29127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.784337 29119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:41.785368 29119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:41.786547 29119 hybrid_clock.cc:648] HybridClock initialized: now 1790526641786529 us; error 35 us; skew 500 ppm
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:41.789163 29119 init.cc:377] Logged in from keytab as kudu/127.15.93.189@KRBTEST.COM (short username kudu)
I20260927 16:30:41.789994 29119 webserver.cc:460] Webserver started at http://127.15.93.189:33679/ using document root <none> and password file <none>
I20260927 16:30:41.790532 29119 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:41.790619 29119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:41.790838 29119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:41.792814 29119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data/instance:
uuid: "7bb99edea9bc4278901a7cdc435a8983"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dac371844fc684f105c98489ef723f1a"
server_key_iv: "84795c8608a6bfab9274bfb09f50fce5"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.793358 29119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal/instance:
uuid: "7bb99edea9bc4278901a7cdc435a8983"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dac371844fc684f105c98489ef723f1a"
server_key_iv: "84795c8608a6bfab9274bfb09f50fce5"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.797005 29119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:30:41.799275 29134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:41.800451 29119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:41.800601 29119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "7bb99edea9bc4278901a7cdc435a8983"
format_stamp: "Formatted at 2026-09-27 16:30:41 on dist-test-slave-grzf"
server_key: "dac371844fc684f105c98489ef723f1a"
server_key_iv: "84795c8608a6bfab9274bfb09f50fce5"
server_key_version: "encryptionkey@0"
I20260927 16:30:41.800709 29119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:41.827185 29119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:41.830101 29119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:41.830312 29119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:41.837260 29119 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.189:37707
I20260927 16:30:41.837287 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.189:37707 every 8 connection(s)
I20260927 16:30:41.838201 29119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
I20260927 16:30:41.840209 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29119
I20260927 16:30:41.840360 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-1/wal/instance
I20260927 16:30:41.840654 15734 external_mini_cluster.cc:1428] Setting key f0e95bae65ecaedb2fe3aea3c5581530
I20260927 16:30:41.841054 29187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:41.844810 29187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.93.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.188@KRBTEST.COM" created.
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:41.858691 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.846723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.15.93.190:34499)
0927 16:30:41.846862 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:41.846866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:41.847841 (+ 975us) server_negotiation.cc:407] Connection header received
0927 16:30:41.848790 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:41.848793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:41.848855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:41.848959 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:41.850247 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.851098 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.852380 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.852646 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.852760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.855133 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:41.855137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:41.855139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:41.855174 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:41.856647 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.857173 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.857178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.857180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.857260 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.857586 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.857591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.857592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.857769 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:41.857874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:41.858401 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 16:30:41.858530 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:30:41.859622 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.847188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34499 (local address 127.0.0.1:47056)
0927 16:30:41.847693 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.847734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:41.848587 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:41.849165 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:41.849171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:41.849476 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:41.850093 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.850108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.851247 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.851283 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.852253 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.852263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.852795 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.852799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.852967 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.853579 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:41.853595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:41.854963 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:41.856766 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.856771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.856786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.857068 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.857351 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.857355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.857357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.857491 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.857872 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:41.857879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:41.858153 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 16:30:41.858339 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 16:30:41.858693 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":165,"threads_started":1}
I20260927 16:30:41.860999 29187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
I20260927 16:30:41.864297 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:36205
--webserver_interface=127.15.93.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:41.865358 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } attempt: 3
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:30:41.872223 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.861510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54428)
0927 16:30:41.861621 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.861635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:41.861707 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:41.862342 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:41.862345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:41.862518 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:41.862616 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.862621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.863946 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.863950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.864850 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.864864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.865700 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.865703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.865797 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.866282 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:41.866293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:41.867811 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:41.870472 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.870475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.870477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.870865 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.871162 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.871165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.871167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.871246 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.871719 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:41.871722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:41.871801 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 16:30:41.871920 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 16:30:41.872048 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:30:41.872267 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.861679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.15.93.189:37707)
0927 16:30:41.862010 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:41.862035 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:41.862053 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:30:41.862085 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:41.862088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:41.862228 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:41.862332 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:41.862723 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.863835 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.865001 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.865440 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.865553 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.867948 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:41.867959 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:41.867976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:41.868002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:41.870374 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.870977 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.870982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.870984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.871067 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.871334 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.871339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.871340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.871613 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:41.871753 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:41.871997 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:30:41.872115 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":123,"threads_started":1}
I20260927 16:30:41.874681 29187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:30:41.875352 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.875114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48188)
0927 16:30:41.875217 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.875287 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 16:30:41.875561 29135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.876654 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.876235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48202)
0927 16:30:41.876548 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.876596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
W20260927 16:30:41.876799 29187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.877329 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.865770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54440)
0927 16:30:41.865969 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.865983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:41.866051 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:41.866730 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:41.866732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:41.866872 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:41.866963 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.866968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.868024 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.868030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.869551 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:41.869565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.870182 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.870187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:41.870330 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.870727 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:41.870749 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:41.873562 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:41.875729 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.875733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.875734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.875985 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.876297 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:41.876301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:41.876303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:41.876388 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:41.876833 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:41.876836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:41.876935 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 16:30:41.877043 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 16:30:41.877177 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 16:30:41.877341 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.865865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.15.93.189:37707)
0927 16:30:41.866496 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:41.866506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:41.866520 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:30:41.866553 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:41.866555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:41.866593 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:41.866681 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:41.867081 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.867913 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.869697 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:41.870051 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:41.870168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:41.873735 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:41.873739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:41.873741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:41.873764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:41.875575 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.876105 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.876110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.876115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.876199 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:41.876495 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:41.876498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:41.876500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:41.876692 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:41.876785 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:41.877090 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 16:30:41.877188 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":420,"threads_started":1}
I20260927 16:30:41.877923 29112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:30:41.878572 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.878349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48218)
0927 16:30:41.878470 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.878519 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:30:41.879135 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.878926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48226)
0927 16:30:41.879053 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.879093 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 16:30:41.879266 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.910419 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 1
I20260927 16:30:41.911250 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.910967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48234)
0927 16:30:41.911113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.911176 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 16:30:41.911996 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.911790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48246)
0927 16:30:41.911896 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.911945 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:30:41.912140 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.926056 29187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 1
I20260927 16:30:41.926853 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.926580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48260)
0927 16:30:41.926714 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.926781 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 16:30:41.927753 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.927517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48266)
0927 16:30:41.927644 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.927699 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 16:30:41.927887 29187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
W20260927 16:30:41.976438 29193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:41.976717 29193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:41.976795 29193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:41.980804 29193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:41.980880 29193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:41.980918 29193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:41.980975 29193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:41.981014 29193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:41.986197 29193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:34499,127.15.93.189:37707,127.15.93.188:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.93.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:41.987259 29193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:41.987293 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 2
I20260927 16:30:41.988108 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.987821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48268)
0927 16:30:41.987963 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.988030 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 16:30:41.988371 29193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:30:41.988816 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.988587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48272)
0927 16:30:41.988706 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.988755 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 16:30:41.988938 29112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.996055 29187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 2
W20260927 16:30:41.996366 29201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:41.996392 29200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.996747 29193 server_base.cc:1034] running on GCE node
I20260927 16:30:41.996965 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.996586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48284)
0927 16:30:41.996825 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.996887 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260927 16:30:41.996950 29203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:41.997629 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:41.997444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48298)
0927 16:30:41.997536 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:41.997577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 16:30:41.997740 29193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260927 16:30:41.997776 29187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36205: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36205: connect: Connection refused (error 111)
I20260927 16:30:41.998875 29193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:42.000046 29193 hybrid_clock.cc:648] HybridClock initialized: now 1790526642000034 us; error 22 us; skew 500 ppm
Sep 27 16:30:41 dist-test-slave-grzf krb5kdc[29031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:42.002799 29193 init.cc:377] Logged in from keytab as kudu/127.15.93.188@KRBTEST.COM (short username kudu)
I20260927 16:30:42.003679 29193 webserver.cc:460] Webserver started at http://127.15.93.188:43025/ using document root <none> and password file <none>
I20260927 16:30:42.004266 29193 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:42.004344 29193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:42.004566 29193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:42.006645 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data/instance:
uuid: "cc00e4ab717f442cadc27e096e1eb5f9"
format_stamp: "Formatted at 2026-09-27 16:30:42 on dist-test-slave-grzf"
server_key: "4d99e195f1f6ab541d42c1d9cbfb1368"
server_key_iv: "0338a3c687267aa7f2b7f90bab46a1c3"
server_key_version: "encryptionkey@0"
I20260927 16:30:42.007212 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal/instance:
uuid: "cc00e4ab717f442cadc27e096e1eb5f9"
format_stamp: "Formatted at 2026-09-27 16:30:42 on dist-test-slave-grzf"
server_key: "4d99e195f1f6ab541d42c1d9cbfb1368"
server_key_iv: "0338a3c687267aa7f2b7f90bab46a1c3"
server_key_version: "encryptionkey@0"
I20260927 16:30:42.010797 29193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:30:42.013182 29210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:42.014346 29193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:30:42.014493 29193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "cc00e4ab717f442cadc27e096e1eb5f9"
format_stamp: "Formatted at 2026-09-27 16:30:42 on dist-test-slave-grzf"
server_key: "4d99e195f1f6ab541d42c1d9cbfb1368"
server_key_iv: "0338a3c687267aa7f2b7f90bab46a1c3"
server_key_version: "encryptionkey@0"
I20260927 16:30:42.014617 29193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:42.047423 29193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:42.050791 29193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:42.051009 29193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:42.057909 29193 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.188:36205
I20260927 16:30:42.057934 29262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.188:36205 every 8 connection(s)
I20260927 16:30:42.058840 29193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
I20260927 16:30:42.061309 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29193
I20260927 16:30:42.061460 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790526488131879-15734-0/minicluster-data/master-2/wal/instance
I20260927 16:30:42.061684 29263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:42.061785 15734 external_mini_cluster.cc:1428] Setting key 67b3cbbfdbdc817e3768ebf3e1d13942
I20260927 16:30:42.066280 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:42.076633 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.063125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.15.93.188:36205)
0927 16:30:42.063575 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.063589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.063627 (+ 38us) server_negotiation.cc:407] Connection header received
0927 16:30:42.064342 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.064360 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.064674 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.065000 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.065774 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.066802 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.067691 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.068345 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.068512 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.071340 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.071345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.071353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.071423 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.073328 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.073804 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.073808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.073813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.073917 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.074212 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.074218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.074221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.074762 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.074989 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.075292 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.075607 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":119,"threads_started":1}
I20260927 16:30:42.077479 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.067989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.15.93.190:34499)
0927 16:30:42.068145 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.068149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.068446 (+ 297us) server_negotiation.cc:407] Connection header received
0927 16:30:42.068684 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.068691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.068751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.068849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.069572 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.070314 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.071347 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.071654 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.071773 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.074089 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.074092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.074094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.074117 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.075850 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.076272 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.076276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.076278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.076359 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.076602 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.076606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.076607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.076830 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.076920 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.077285 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.077375 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:30:42.077540 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.067933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:34499 (local address 127.0.0.1:47062)
0927 16:30:42.068329 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:42.068379 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:42.068526 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:42.068911 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:42.068917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:42.069321 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:42.069446 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.069451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.070445 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.070451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.071224 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.071234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.071776 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.071778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.071940 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.072401 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:42.072420 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:42.073945 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:42.075954 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.075961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.075965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.076179 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.076445 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.076447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.076448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.076515 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.076934 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:42.076940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:42.077065 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 16:30:42.077226 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 16:30:42.077379 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":146,"threads_started":1}
I20260927 16:30:42.079133 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:42.087785 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.079613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:37707 (local address 127.0.0.1:54444)
0927 16:30:42.079858 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:42.079872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:42.079964 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:42.080201 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:42.080204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:42.080379 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:42.080463 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.080468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.081352 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.081356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.082112 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.082120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.082645 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.082651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.082723 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.083000 (+ 277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:42.083015 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:42.084324 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:42.086217 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.086224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.086227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.086459 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.086768 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.086771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.086773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.086875 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.087279 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:42.087282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:42.087365 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 16:30:42.087489 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 16:30:42.087615 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 16:30:42.087867 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.079722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.15.93.189:37707)
0927 16:30:42.079830 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.079834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.079918 (+ 84us) server_negotiation.cc:407] Connection header received
0927 16:30:42.080055 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.080057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.080093 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.080156 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.080572 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.081255 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.082206 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.082534 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.082614 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.084458 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.084462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.084465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.084493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.086108 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.086554 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.086558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.086560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.086657 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.086959 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.086962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.086964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.087178 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.087263 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.087555 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.087699 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:30:42.087982 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.079193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.15.93.190:34499)
0927 16:30:42.079362 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.079398 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.079411 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:30:42.079444 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.079447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.079492 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.079567 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.080340 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.081008 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.082025 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.082294 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.082389 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.084996 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.084999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.085001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.085021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.086325 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.086768 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.086771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.086773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.086867 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.087139 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.087141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.087142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.087339 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.087601 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.087694 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.087794 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 16:30:42.088176 29112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 3
I20260927 16:30:42.088928 29187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } attempt: 3
I20260927 16:30:42.089061 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:30:42 dist-test-slave-grzf krb5kdc[29031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:30:42.099181 29115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.088504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48306)
0927 16:30:42.088671 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:42.088687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:42.088770 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:42.089274 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:42.089277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:42.089447 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:42.089535 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.089543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.090856 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.090860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.091666 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.091683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.092253 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.092257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.092337 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.092812 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:42.092825 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:42.094259 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:42.097268 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.097274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.097278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.097532 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.098106 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.098112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.098117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.098218 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.098667 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:42.098671 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:42.098769 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 16:30:42.098858 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:30:42.098989 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 16:30:42.099753 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.088539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.15.93.188:36205)
0927 16:30:42.088668 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.088672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.088749 (+ 77us) server_negotiation.cc:407] Connection header received
0927 16:30:42.088865 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.088871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.089147 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.089889 (+ 742us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.089971 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.090737 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.091804 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.092132 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.092522 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.094387 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.094392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.094394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.094419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.097154 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.097704 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.097708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.097710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.098006 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.098325 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.098331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.098333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.098543 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.099356 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.099459 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.099578 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":53}
I20260927 16:30:42.100385 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.089275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48320)
0927 16:30:42.089430 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:42.089442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:42.089508 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:42.090249 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:42.090252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:42.090414 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:42.090500 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.090505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.091758 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.091764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.092565 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.092576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.093213 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.093219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.093366 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.093812 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:42.093826 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:42.096003 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:42.098661 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.098664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.098666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.099068 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.099396 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.099401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.099403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.099488 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.099917 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:42.099919 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:42.100013 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 16:30:42.100116 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 16:30:42.100236 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":18}
I20260927 16:30:42.100813 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.089680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.15.93.188:36205)
0927 16:30:42.090026 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.090035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.090045 (+ 10us) server_negotiation.cc:407] Connection header received
0927 16:30:42.090071 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.090074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.090119 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.090595 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.090758 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.091626 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.092705 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.093067 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.093353 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.096386 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.096391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.096396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.096431 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.098540 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.099180 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.099183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.099185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.099267 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.099601 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.099604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.099605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.099833 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.099924 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.100563 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.100673 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":5504,"thread_start_us":116,"threads_started":1}
I20260927 16:30:42.101155 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.089453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36205 (local address 127.0.0.1:48324)
0927 16:30:42.089628 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:42.089641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:42.089699 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:42.090261 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:42.090264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:42.090431 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:42.090514 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.090520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.092151 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.092154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.093166 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:42.093175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.093762 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.093766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:42.093838 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.094227 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:42.094239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:42.096482 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:42.098891 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.098893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.098897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.099167 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.099790 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:42.099794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:42.099799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:42.099885 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:42.100716 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:42.100719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:42.100809 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 16:30:42.100887 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:30:42.101010 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 16:30:42.102005 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.089660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.15.93.189:37707)
0927 16:30:42.089990 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.089994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.090266 (+ 272us) server_negotiation.cc:407] Connection header received
0927 16:30:42.090306 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.090309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.090348 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.090416 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.091273 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.092037 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.093112 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.093659 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.094402 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.097061 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.097064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.097066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.097089 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.099232 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.100043 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.100046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.100048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.100136 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.100689 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.100695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.100700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.100897 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.101082 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.101737 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.101852 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":257,"spinlock_wait_cycles":1920}
I20260927 16:30:42.102097 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.089764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.15.93.188:36205)
0927 16:30:42.090039 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.090047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.090058 (+ 11us) server_negotiation.cc:407] Connection header received
0927 16:30:42.090093 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.090097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.090144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.090622 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.091224 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.092042 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.093274 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.093645 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.094078 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.096640 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.096645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.096648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.096673 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.098775 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.099583 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.099586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.099588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.099691 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.100194 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.100200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.100201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.100399 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.101770 (+ 1371us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.101840 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.101932 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260927 16:30:42.102967 29112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: Bootstrap starting.
I20260927 16:30:42.105122 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:30:42.105441 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: Bootstrap starting.
I20260927 16:30:42.109566 29187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: Bootstrap starting.
I20260927 16:30:42.109617 29112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:42.110442 29112 log.cc:824] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:42.111315 29263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:42.111517 29187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:42.112349 29187 log.cc:824] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:42.112411 29263 log.cc:824] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:42.112495 29112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: No bootstrap required, opened a new log
I20260927 16:30:42.114535 29187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: No bootstrap required, opened a new log
I20260927 16:30:42.114979 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: No bootstrap required, opened a new log
I20260927 16:30:42.116570 29116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.107071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.15.93.190:34499)
0927 16:30:42.107191 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.107196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.107887 (+ 691us) server_negotiation.cc:407] Connection header received
0927 16:30:42.108000 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.108006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.108059 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.108145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.108991 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.109696 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.110794 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.111087 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.111189 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.113034 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.113041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.113046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.113074 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.114746 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.115220 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.115224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.115226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.115308 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.115653 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.115656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.115658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.115855 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.115965 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.116318 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.116425 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:30:42.116888 29112 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.117267 29112 raft_consensus.cc:374] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:42.117328 29112 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb9240c4887402bb1339d8153fa63d4, State: Initialized, Role: FOLLOWER
I20260927 16:30:42.118091 29112 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.119341 29187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.119612 29112 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:30:42.119701 29065 tablet.cc:2369] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:30:42.119649 29263 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.119854 29263 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:42.119903 29263 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc00e4ab717f442cadc27e096e1eb5f9, State: Initialized, Role: FOLLOWER
I20260927 16:30:42.120237 29187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:42.120340 29187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb99edea9bc4278901a7cdc435a8983, State: Initialized, Role: FOLLOWER
I20260927 16:30:42.120363 29263 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.120841 29187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.121966 29263 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:42.122080 29286 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.122191 29286 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.122522 29187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:30:42.125211 29293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:30:42.125286 29293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:30:42.127029 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.107457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.15.93.189:37707)
0927 16:30:42.107557 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.107561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.117906 (+ 10345us) server_negotiation.cc:407] Connection header received
0927 16:30:42.118073 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.118076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.118119 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.118167 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.119058 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.119797 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.120837 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.121155 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.121233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.123340 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.123345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.123350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.123402 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.125062 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.125702 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.125708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.125710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.125793 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.126121 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.126124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.126126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.126325 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.126420 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.126791 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.126895 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:30:42.124795 29044 master_runner.cc:186] Error getting master registration for 127.15.93.189:37707: OK, instance_id { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" instance_seqno: 1790526641835201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:30:42.128618 29044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:30:42.129345 29305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:30:42.129534 29305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:30:42.129931 29287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.130089 29287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.132895 29280 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.133019 29280 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.130504 29119 master_runner.cc:186] Error getting master registration for 127.15.93.188:36205: OK, instance_id { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" instance_seqno: 1790526642056017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:30:42.134004 29119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:30:42.139243 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:42.107457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.15.93.188:36205)
0927 16:30:42.107637 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:42.107640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:42.127578 (+ 19938us) server_negotiation.cc:407] Connection header received
0927 16:30:42.127779 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:42.127783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:42.127830 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:42.127880 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:42.128690 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.129441 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.130459 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:42.130808 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:42.130884 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:42.135413 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:42.135417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:42.135420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:42.135446 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:42.137177 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.137677 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.137681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.137682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.137771 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:42.138080 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:42.138083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:42.138085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:42.138308 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:42.138450 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:42.138892 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 16:30:42.139025 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 16:30:42.142377 29193 master_runner.cc:263] No leader master found from master cc00e4ab717f442cadc27e096e1eb5f9
I20260927 16:30:42.142445 29193 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 16:30:42.153573 29317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:30:42.153671 29317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 16:30:42.302488 29216 tablet.cc:2369] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:30:42.332093 29140 tablet.cc:2369] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:30:42.455642 29280 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:42.455813 29280 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.456807 29280 leader_election.cc:288] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bb99edea9bc4278901a7cdc435a8983 (127.15.93.189:37707), cc00e4ab717f442cadc27e096e1eb5f9 (127.15.93.188:36205)
I20260927 16:30:42.457315 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9240c4887402bb1339d8153fa63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb99edea9bc4278901a7cdc435a8983" is_pre_election: true
I20260927 16:30:42.457325 29237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9240c4887402bb1339d8153fa63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" is_pre_election: true
I20260927 16:30:42.457607 29161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb9240c4887402bb1339d8153fa63d4 in term 0.
I20260927 16:30:42.457665 29237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb9240c4887402bb1339d8153fa63d4 in term 0.
I20260927 16:30:42.458074 29061 leader_election.cc:302] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7bb99edea9bc4278901a7cdc435a8983, afb9240c4887402bb1339d8153fa63d4, cc00e4ab717f442cadc27e096e1eb5f9; no voters:
I20260927 16:30:42.458380 29280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:30:42.458480 29280 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:30:42.458559 29280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:42.459745 29280 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.460067 29280 leader_election.cc:288] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [CANDIDATE]: Term 1 election: Requested vote from peers 7bb99edea9bc4278901a7cdc435a8983 (127.15.93.189:37707), cc00e4ab717f442cadc27e096e1eb5f9 (127.15.93.188:36205)
I20260927 16:30:42.460422 29237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9240c4887402bb1339d8153fa63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc00e4ab717f442cadc27e096e1eb5f9"
I20260927 16:30:42.460438 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb9240c4887402bb1339d8153fa63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb99edea9bc4278901a7cdc435a8983"
I20260927 16:30:42.460569 29161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:42.460525 29237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:42.461539 29161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9240c4887402bb1339d8153fa63d4 in term 1.
I20260927 16:30:42.461566 29237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9240c4887402bb1339d8153fa63d4 in term 1.
I20260927 16:30:42.461836 29060 leader_election.cc:302] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb9240c4887402bb1339d8153fa63d4, cc00e4ab717f442cadc27e096e1eb5f9; no voters:
I20260927 16:30:42.462020 29280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:42.462239 29280 raft_consensus.cc:686] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [term 1 LEADER]: Becoming Leader. State: Replica: afb9240c4887402bb1339d8153fa63d4, State: Running, Role: LEADER
I20260927 16:30:42.462550 29280 consensus_queue.cc:237] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } }
I20260927 16:30:42.464437 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb9240c4887402bb1339d8153fa63d4. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.464556 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:42.465013 29321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:42.465838 29321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:42.470124 29237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [term 1 FOLLOWER]: Refusing update from remote peer afb9240c4887402bb1339d8153fa63d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:42.470129 29161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [term 1 FOLLOWER]: Refusing update from remote peer afb9240c4887402bb1339d8153fa63d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:42.470805 29280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:42.470989 29319 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:42.474704 29286 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb9240c4887402bb1339d8153fa63d4. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.474823 29286 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.476346 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.476462 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:42.476727 29286 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.476818 29286 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.477593 29287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb9240c4887402bb1339d8153fa63d4. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.477705 29287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.477849 29322 mvcc.cc:205] Tried to move back new op lower bound from 7333997127551332352 to 7333997127529119744. Current Snapshot: MvccSnapshot[applied={T|T < 7333997127551332352}]
I20260927 16:30:42.478726 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.478828 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:42.479455 29326 mvcc.cc:205] Tried to move back new op lower bound from 7333997127551332352 to 7333997127529119744. Current Snapshot: MvccSnapshot[applied={T|T < 7333997127551332352}]
I20260927 16:30:42.479503 29321 catalog_manager.cc:1347] Generated new cluster ID: ee67d4b46ec04aff8899a05d65f994d3
I20260927 16:30:42.479553 29321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:42.479653 29287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afb9240c4887402bb1339d8153fa63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9240c4887402bb1339d8153fa63d4" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 34499 } } peers { permanent_uuid: "7bb99edea9bc4278901a7cdc435a8983" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 37707 } } peers { permanent_uuid: "cc00e4ab717f442cadc27e096e1eb5f9" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36205 } } }
I20260927 16:30:42.479758 29287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:30:42.491889 29321 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:30:42.492873 29321 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:42.499030 29321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afb9240c4887402bb1339d8153fa63d4: Generated new TSK 0
I20260927 16:30:42.500094 29321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:30:43.130894 29305 catalog_manager.cc:1259] Loaded cluster ID: ee67d4b46ec04aff8899a05d65f994d3
I20260927 16:30:43.130980 29305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: loading cluster ID for follower catalog manager: success
I20260927 16:30:43.132691 29305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: acquiring CA information for follower catalog manager: success
I20260927 16:30:43.133857 29305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bb99edea9bc4278901a7cdc435a8983: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:30:43.154870 29317 catalog_manager.cc:1259] Loaded cluster ID: ee67d4b46ec04aff8899a05d65f994d3
I20260927 16:30:43.154945 29317 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: loading cluster ID for follower catalog manager: success
I20260927 16:30:43.156711 29317 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: acquiring CA information for follower catalog manager: success
I20260927 16:30:43.157819 29317 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc00e4ab717f442cadc27e096e1eb5f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:30:43.552661 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29044
I20260927 16:30:43.558960 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29119
I20260927 16:30:43.564677 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29193
2026-09-27T16:30:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3264 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:30:43 dist-test-slave-grzf krb5kdc[29337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:30:43 dist-test-slave-grzf krb5kdc[29337](info): set up 2 sockets
Sep 27 16:30:43 dist-test-slave-grzf krb5kdc[29337](info): commencing operation
krb5kdc: starting...
W20260927 16:30:44.591570 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:30:44 dist-test-slave-grzf krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:30:44.709890 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42545
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:33825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:42545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:44.816991 29350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:44.817225 29350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:44.817281 29350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:44.821099 29350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:44.821138 29350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:44.821156 29350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:44.821172 29350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:44.821187 29350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:44.826310 29350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:33825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:42545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:44.827221 29350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:44.828150 29350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:44.836128 29355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:44.836097 29356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:44.836115 29358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:44.836570 29350 server_base.cc:1034] running on GCE node
I20260927 16:30:44.837117 29350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:44.838207 29350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:44.839361 29350 hybrid_clock.cc:648] HybridClock initialized: now 1790526644839334 us; error 40 us; skew 500 ppm
Sep 27 16:30:44 dist-test-slave-grzf krb5kdc[29337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526644, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:44.842177 29350 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:44.843029 29350 webserver.cc:460] Webserver started at http://127.15.93.190:46011/ using document root <none> and password file <none>
I20260927 16:30:44.843637 29350 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:44.843701 29350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:44.843943 29350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:44.845911 29350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "8ccd2891f309487bb74587f2cea046fe"
format_stamp: "Formatted at 2026-09-27 16:30:44 on dist-test-slave-grzf"
server_key: "b17725e4a61d9cfab60837256d9b5514"
server_key_iv: "f2449e3baab108fee8b515a49dfb8297"
server_key_version: "encryptionkey@0"
I20260927 16:30:44.846462 29350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "8ccd2891f309487bb74587f2cea046fe"
format_stamp: "Formatted at 2026-09-27 16:30:44 on dist-test-slave-grzf"
server_key: "b17725e4a61d9cfab60837256d9b5514"
server_key_iv: "f2449e3baab108fee8b515a49dfb8297"
server_key_version: "encryptionkey@0"
I20260927 16:30:44.850072 29350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:30:44.852372 29365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:44.853467 29350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:30:44.853612 29350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "8ccd2891f309487bb74587f2cea046fe"
format_stamp: "Formatted at 2026-09-27 16:30:44 on dist-test-slave-grzf"
server_key: "b17725e4a61d9cfab60837256d9b5514"
server_key_iv: "f2449e3baab108fee8b515a49dfb8297"
server_key_version: "encryptionkey@0"
I20260927 16:30:44.853718 29350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:44.896294 29350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:44.899662 29350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:44.899865 29350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:44.906803 29350 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:42545
I20260927 16:30:44.906832 29417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:42545 every 8 connection(s)
I20260927 16:30:44.907732 29350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:44.910316 29418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:44.915611 29418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: Bootstrap starting.
I20260927 16:30:44.915987 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29350
I20260927 16:30:44.916107 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:30:44.916406 15734 external_mini_cluster.cc:1428] Setting key 9b5d0fce8c37b6d09c221d0f47b17f3e
I20260927 16:30:44.918195 29418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:44.918814 29418 log.cc:824] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:44.920495 29418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: No bootstrap required, opened a new log
I20260927 16:30:44.923849 29418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } }
I20260927 16:30:44.924042 29418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:44.924120 29418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccd2891f309487bb74587f2cea046fe, State: Initialized, Role: FOLLOWER
I20260927 16:30:44.924582 29418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } }
I20260927 16:30:44.924727 29418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:44.924813 29418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 27 16:30:44 dist-test-slave-grzf krb5kdc[29337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:44.924927 29418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:44.925927 29418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } }
I20260927 16:30:44.926261 29418 leader_election.cc:302] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ccd2891f309487bb74587f2cea046fe; no voters:
I20260927 16:30:44.926579 29418 leader_election.cc:288] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:30:44.926710 29423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:44.926906 29423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [term 1 LEADER]: Becoming Leader. State: Replica: 8ccd2891f309487bb74587f2cea046fe, State: Running, Role: LEADER
I20260927 16:30:44.927256 29423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } }
I20260927 16:30:44.927817 29418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:44.928507 29425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ccd2891f309487bb74587f2cea046fe. Latest consensus state: current_term: 1 leader_uuid: "8ccd2891f309487bb74587f2cea046fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } } }
I20260927 16:30:44.929277 29425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:44.929128 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ccd2891f309487bb74587f2cea046fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ccd2891f309487bb74587f2cea046fe" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 42545 } } }
I20260927 16:30:44.929596 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:44.931360 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:44.917611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.15.93.190:42545)
0927 16:30:44.918000 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:44.918014 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:44.918047 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:30:44.918714 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:44.918733 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:44.919132 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:44.919533 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:44.920305 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:44.921320 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:44.922190 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:44.922654 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:44.922833 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:44.925479 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:44.925485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:44.925496 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:44.925521 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:44.927607 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:44.928092 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:44.928104 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:44.928110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:44.928242 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:44.928548 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:44.928555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:44.928559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:44.929100 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:44.929290 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:44.930177 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 16:30:44.930499 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
W20260927 16:30:44.932760 29438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:30:44.932847 29438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:30:44.932917 29434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:44.933740 29434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:44.934338 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:30:44.938570 29434 catalog_manager.cc:1347] Generated new cluster ID: 82074159088a406581ae224f8f650733
I20260927 16:30:44.938628 29434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:44.949606 29434 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:30:44.950783 29434 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:44.957057 29434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ccd2891f309487bb74587f2cea046fe: Generated new TSK 0
I20260927 16:30:44.957845 29434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:30:44.990270 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29350
2026-09-27T16:30:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:30:45 dist-test-slave-grzf krb5kdc[29445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:30:45 dist-test-slave-grzf krb5kdc[29445](info): set up 2 sockets
Sep 27 16:30:45 dist-test-slave-grzf krb5kdc[29445](info): commencing operation
krb5kdc: starting...
W20260927 16:30:46.015588 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:30:46.133523 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:46.246887 29458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:46.247138 29458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:46.247211 29458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:46.251169 29458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:46.251226 29458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:46.251245 29458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:46.251262 29458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:46.251323 29458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:46.256515 29458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:46.257575 29458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:46.258585 29458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:46.266712 29466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.266718 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.266708 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.267017 29458 server_base.cc:1034] running on GCE node
I20260927 16:30:46.267640 29458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:46.268618 29458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:46.269771 29458 hybrid_clock.cc:648] HybridClock initialized: now 1790526646269744 us; error 39 us; skew 500 ppm
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:46.278678 29458 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:46.279750 29458 webserver.cc:460] Webserver started at http://127.15.93.190:42069/ using document root <none> and password file <none>
I20260927 16:30:46.280309 29458 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:46.280370 29458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:46.280601 29458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:46.282829 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "a5cf9d69c1354c47a1dba03bfc61f63d"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "7f4e292f891c2f2a0f81fec6d9c9eb65"
server_key_iv: "4ead027593a30bdaf09a8d8ab30e8ef2"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.283605 29458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "a5cf9d69c1354c47a1dba03bfc61f63d"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "7f4e292f891c2f2a0f81fec6d9c9eb65"
server_key_iv: "4ead027593a30bdaf09a8d8ab30e8ef2"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.287806 29458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 16:30:46.290117 29473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.291285 29458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:46.291458 29458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "a5cf9d69c1354c47a1dba03bfc61f63d"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "7f4e292f891c2f2a0f81fec6d9c9eb65"
server_key_iv: "4ead027593a30bdaf09a8d8ab30e8ef2"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.291568 29458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:46.309890 29458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:46.312810 29458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:46.313015 29458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:46.319650 29458 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:33681
I20260927 16:30:46.319893 29525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:33681 every 8 connection(s)
I20260927 16:30:46.320647 29458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:46.323292 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:46.328575 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap starting.
I20260927 16:30:46.329957 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29458
I20260927 16:30:46.330116 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:30:46.330382 15734 external_mini_cluster.cc:1428] Setting key 55640305a336050025abd4ecf3e3c14f
I20260927 16:30:46.331095 29526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:46.332001 29526 log.cc:824] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:46.333894 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: No bootstrap required, opened a new log
I20260927 16:30:46.337311 29526 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:46.337553 29526 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:46.337642 29526 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Initialized, Role: FOLLOWER
I20260927 16:30:46.338078 29526 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:46.338229 29526 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:46.338315 29526 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:30:46.338429 29526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:46.339677 29526 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:46.340030 29526 leader_election.cc:302] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5cf9d69c1354c47a1dba03bfc61f63d; no voters:
I20260927 16:30:46.340386 29526 leader_election.cc:288] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:30:46.340495 29531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:46.340718 29531 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 LEADER]: Becoming Leader. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Running, Role: LEADER
I20260927 16:30:46.341079 29531 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:46.341814 29526 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:46.345201 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:46.346987 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:46.347009 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5cf9d69c1354c47a1dba03bfc61f63d. Latest consensus state: current_term: 1 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:46.347080 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:46.347841 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:46.331532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.15.93.190:33681)
0927 16:30:46.331957 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:46.331971 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:46.332005 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:30:46.332893 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:46.332912 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:46.333233 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:46.333569 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:46.334363 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.335335 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.336237 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.336730 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.336901 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:46.339889 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:46.339894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:46.339906 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:46.339931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:46.343062 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.343620 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.343627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.343633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.343767 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.344083 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.344087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.344088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.344630 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:46.344845 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:46.345789 (+ 944us) server_negotiation.cc:299] Negotiation successful
0927 16:30:46.346414 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
W20260927 16:30:46.349062 29546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:30:46.349146 29546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:30:46.349221 29548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:46.349897 29548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:46.355203 29548 catalog_manager.cc:1347] Generated new cluster ID: c17869e283d04e12989f160abcc100df
I20260927 16:30:46.355505 29548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:46.388267 29548 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:30:46.389827 29548 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:46.401306 29548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Generated new TSK 0
I20260927 16:30:46.401945 29548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:30:46.433947 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:46.563293 29552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:46.563581 29552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:46.563670 29552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:46.567677 29552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:46.567727 29552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:46.567785 29552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:30:46.572753 29552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:46.573657 29552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:46.574585 29552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:46.583464 29558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.583472 29560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.583698 29552 server_base.cc:1034] running on GCE node
W20260927 16:30:46.583559 29557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.584182 29552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:46.584766 29552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:46.585919 29552 hybrid_clock.cc:648] HybridClock initialized: now 1790526646585891 us; error 41 us; skew 500 ppm
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:46.588665 29552 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:30:46.589394 29552 webserver.cc:460] Webserver started at http://127.15.93.129:44533/ using document root <none> and password file <none>
I20260927 16:30:46.589881 29552 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:46.589936 29552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:46.590088 29552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:46.591993 29552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "4745e9da5a6547ef90f2c86756c71e92"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "77554f6c6cc408a1f7423236845a6c73"
server_key_iv: "695b2285db1562346137f63a6e926a41"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.592476 29552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "4745e9da5a6547ef90f2c86756c71e92"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "77554f6c6cc408a1f7423236845a6c73"
server_key_iv: "695b2285db1562346137f63a6e926a41"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.595873 29552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:30:46.598026 29567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.599181 29552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:46.599295 29552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "4745e9da5a6547ef90f2c86756c71e92"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "77554f6c6cc408a1f7423236845a6c73"
server_key_iv: "695b2285db1562346137f63a6e926a41"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.599464 29552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:46.614048 29552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:46.617221 29552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:46.617411 29552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:46.617954 29552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:46.618717 29552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:46.618760 29552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.618825 29552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:46.618855 29552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.627274 29552 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:39469
I20260927 16:30:46.627300 29680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:39469 every 8 connection(s)
I20260927 16:30:46.628100 29552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:30:46.630074 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29552
I20260927 16:30:46.630203 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:30:46.630489 15734 external_mini_cluster.cc:1428] Setting key 5d7f654646ee228bdd68181cae704659
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:30:46.643714 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:46.629831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:42709 (local address 127.15.93.190:33681)
0927 16:30:46.629999 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:46.630004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:46.630888 (+ 884us) server_negotiation.cc:407] Connection header received
0927 16:30:46.632602 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:46.632605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:46.632644 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:46.632711 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:46.634127 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.634974 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.636771 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.636993 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.637072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:46.639795 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:46.639798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:46.639800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:46.639821 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:46.641366 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.641880 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.641884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.641885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.641966 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.642445 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.642449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.642450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.642653 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:46.642752 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:46.643481 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 16:30:46.643594 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 16:30:46.644761 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:46.630253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:42709)
0927 16:30:46.630750 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:46.630804 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:46.632391 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:46.632881 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:46.632892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:46.633395 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:46.633987 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:46.634001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.635099 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.635106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:46.636657 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:46.636669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.637150 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.637156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:46.637394 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:46.637991 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:46.638018 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:46.639638 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:46.641503 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:46.641508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:46.641522 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:46.641773 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:46.642057 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:46.642060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:46.642063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:46.642342 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:46.642763 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:46.642771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:46.643098 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 16:30:46.643306 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 16:30:46.643696 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":104,"threads_started":1}
I20260927 16:30:46.645983 29681 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:46.646237 29681 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:46.646848 29681 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:46.648475 29490 ts_manager.cc:194] Registered new tserver with Master: 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129:39469)
I20260927 16:30:46.649574 29490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:42709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:30:46.658067 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:46.764321 29686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:46.764567 29686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:46.764608 29686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:46.768610 29686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:46.768666 29686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:46.768744 29686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:30:46.773826 29686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:46.774878 29686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:46.775991 29686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:46.785254 29694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.785265 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.785260 29692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.785297 29686 server_base.cc:1034] running on GCE node
I20260927 16:30:46.785948 29686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:46.786557 29686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:46.787719 29686 hybrid_clock.cc:648] HybridClock initialized: now 1790526646787706 us; error 25 us; skew 500 ppm
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:46.790414 29686 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:30:46.791226 29686 webserver.cc:460] Webserver started at http://127.15.93.130:45537/ using document root <none> and password file <none>
I20260927 16:30:46.791831 29686 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:46.791926 29686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:46.792162 29686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:46.794188 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "03be8534a2de44c7b82726b162e4bf35"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "1cb688f39a4358c7a853170ac15077fe"
server_key_iv: "e7e68fab44596d112b6d3a71d3a3f29f"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.794754 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "03be8534a2de44c7b82726b162e4bf35"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "1cb688f39a4358c7a853170ac15077fe"
server_key_iv: "e7e68fab44596d112b6d3a71d3a3f29f"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.798282 29686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:30:46.800621 29701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.801780 29686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:30:46.801918 29686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "03be8534a2de44c7b82726b162e4bf35"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "1cb688f39a4358c7a853170ac15077fe"
server_key_iv: "e7e68fab44596d112b6d3a71d3a3f29f"
server_key_version: "encryptionkey@0"
I20260927 16:30:46.802026 29686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:46.814869 29686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:46.818156 29686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:46.818360 29686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:46.818941 29686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:46.819753 29686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:46.819823 29686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.819892 29686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:46.819936 29686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:46.828498 29686 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:45707
I20260927 16:30:46.828558 29814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:45707 every 8 connection(s)
I20260927 16:30:46.829408 29686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:30:46.833865 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29686
I20260927 16:30:46.833995 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:30:46.834290 15734 external_mini_cluster.cc:1428] Setting key 369ca2d9b06972ed82793d20eb7a5dd4
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:30:46.844295 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:46.831203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:43973 (local address 127.15.93.190:33681)
0927 16:30:46.831337 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:46.831342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:46.832271 (+ 929us) server_negotiation.cc:407] Connection header received
0927 16:30:46.833683 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:46.833686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:46.833729 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:46.833802 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:46.835300 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.835941 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.837710 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.838091 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.838199 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:46.840609 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:46.840613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:46.840615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:46.840644 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:46.842159 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.842700 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.842707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.842708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.842786 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:46.843139 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:46.843144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:46.843146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:46.843320 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:46.843468 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:46.844007 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 16:30:46.844131 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:30:46.845193 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:46.831654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:43973)
0927 16:30:46.832093 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:46.832151 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:46.833436 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:46.834027 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:46.834037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:46.834550 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:46.835165 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:46.835183 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.836059 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.836063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:46.837560 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:46.837575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:46.838228 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:46.838231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:46.838366 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:46.838984 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:46.839000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:46.840450 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:46.842271 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:46.842277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:46.842292 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:46.842537 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:46.842903 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:46.842905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:46.842907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:46.843037 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:46.843482 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:46.843490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:46.843772 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 16:30:46.843953 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:30:46.844281 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:30:46.846431 29815 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:46.846621 29815 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:46.847193 29815 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:46.848309 29490 ts_manager.cc:194] Registered new tserver with Master: 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:46.848881 29490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:43973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:30:46.860505 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:46.969792 29820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:46.970057 29820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:46.970103 29820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:46.974133 29820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:46.974195 29820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:46.974253 29820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:30:46.979486 29820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:46.980594 29820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:46.981653 29820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:46.991127 29825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:46.991142 29826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.991259 29820 server_base.cc:1034] running on GCE node
W20260927 16:30:46.991192 29828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:46.991866 29820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:46.992515 29820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:46.993669 29820 hybrid_clock.cc:648] HybridClock initialized: now 1790526646993654 us; error 27 us; skew 500 ppm
Sep 27 16:30:46 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:46.996495 29820 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:30:46.997375 29820 webserver.cc:460] Webserver started at http://127.15.93.131:37449/ using document root <none> and password file <none>
I20260927 16:30:46.997921 29820 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:46.998009 29820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:46.998242 29820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:30:47.000298 29820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "a200477dd3ff4919ad3d81c41587cfd3"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "b33d9a80a6615f9526a43bd4b061463c"
server_key_iv: "ff4b3fa442363e117561ec35c79f577d"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.000837 29820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "a200477dd3ff4919ad3d81c41587cfd3"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "b33d9a80a6615f9526a43bd4b061463c"
server_key_iv: "ff4b3fa442363e117561ec35c79f577d"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.004521 29820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:30:47.006969 29835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.008160 29820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:30:47.008291 29820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "a200477dd3ff4919ad3d81c41587cfd3"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "b33d9a80a6615f9526a43bd4b061463c"
server_key_iv: "ff4b3fa442363e117561ec35c79f577d"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.008401 29820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:47.053452 29820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:47.056844 29820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:47.057051 29820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:47.057660 29820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:47.058494 29820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:47.058563 29820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.058632 29820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:47.058687 29820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.067914 29820 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:43651
I20260927 16:30:47.067932 29948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:43651 every 8 connection(s)
I20260927 16:30:47.068835 29820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:30:47.076758 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29820
I20260927 16:30:47.076891 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:30:47.077154 15734 external_mini_cluster.cc:1428] Setting key 9917b0aa8c4b75bf0c8e11fe9a4b6c16
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:47.082294 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.070615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:56403 (local address 127.15.93.190:33681)
0927 16:30:47.070770 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.070774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.071709 (+ 935us) server_negotiation.cc:407] Connection header received
0927 16:30:47.073012 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.073015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.073062 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.073143 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.074258 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.074791 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.075902 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.076123 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.076202 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.078942 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.078945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.078947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.078974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.080468 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.080949 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.080951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.080953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.081027 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.081340 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.081342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.081343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.081518 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.081590 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.082062 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.082172 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:30:47.083145 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:56403)
0927 16:30:47.071550 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.071606 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.072824 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.073300 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.073307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.073662 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:47.074133 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.074146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.074898 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.074900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.075796 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.075810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.076248 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.076251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.076389 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.077084 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:47.077110 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:47.078744 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:47.080569 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.080574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.080587 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.080860 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.081111 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.081113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.081114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.081250 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.081611 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:47.081617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:47.081855 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.082019 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.082281 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260927 16:30:47.084306 29949 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:47.084518 29949 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:47.084959 29949 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:47.085907 29490 ts_manager.cc:194] Registered new tserver with Master: a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651)
I20260927 16:30:47.086481 29490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:56403
I20260927 16:30:47.091238 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:30:47.100463 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.093220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.15.93.190:33681)
0927 16:30:47.093339 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.093343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.093531 (+ 188us) server_negotiation.cc:407] Connection header received
0927 16:30:47.093641 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.093643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.093679 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.093743 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.094459 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.094949 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.095867 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.096071 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.096151 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.097379 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.097382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.097384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.097405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.098833 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.099198 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.099201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.099202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.099270 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.099574 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.099578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.099580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.099808 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.099958 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.100179 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.100291 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:30:47.103060 29490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:30:47.106348 29490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:30:47.120762 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.115521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.15.93.130:45707)
0927 16:30:47.115821 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.115832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.115855 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:30:47.115937 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.115944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.116165 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.116597 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:47.116769 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.117521 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.119467 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120010 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.120148 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.120305 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.120511 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.120614 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260927 16:30:47.120813 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.114366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43651 (local address 127.0.0.1:34504)
0927 16:30:47.115086 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.115135 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.115262 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.116033 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.116040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.116065 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:47.116211 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.116220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.117698 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.117701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.119206 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.119217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120248 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.120252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.120362 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.120391 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.120441 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.120604 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":117,"thread_start_us":76,"threads_started":1}
I20260927 16:30:47.120813 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.114454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:39469 (local address 127.0.0.1:44114)
0927 16:30:47.115206 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.115221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.115312 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.116654 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.116659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.116676 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:47.116759 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.116765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.118028 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.118033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.119090 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.119115 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120424 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.120430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.120510 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.120534 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.120613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.120672 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260927 16:30:47.120945 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.115143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.15.93.129:39469)
0927 16:30:47.116067 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.116082 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.116102 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:30:47.116169 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.116174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.116337 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.116472 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:47.116888 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.117909 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.119221 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120107 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.120225 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.120367 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.120689 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.120783 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":82,"threads_started":1}
I20260927 16:30:47.121181 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.114722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.0.0.1:48480)
0927 16:30:47.115267 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.115280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.115346 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.116323 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.116326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.116342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:47.116453 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.116460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.117653 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.117660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.119206 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.119217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120146 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.120150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.120252 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.120330 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.120476 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.120614 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":195,"thread_start_us":92,"threads_started":1}
I20260927 16:30:47.121485 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.114475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.15.93.131:43651)
0927 16:30:47.115597 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.115608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.115626 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:30:47.115689 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.115693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.115848 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.116369 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:47.116526 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.117547 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.119327 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.120114 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.121063 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.121209 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.121284 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.121380 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":117,"threads_started":1}
I20260927 16:30:47.124003 29883 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 (DEFAULT_TABLE table=test-table [id=5eb15641442e47fdabc12a350362e77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:47.124949 29883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b5c33e1fa44bfcbc1cd79b73559356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:47.125006 29615 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 (DEFAULT_TABLE table=test-table [id=5eb15641442e47fdabc12a350362e77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:47.126021 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b5c33e1fa44bfcbc1cd79b73559356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:47.130162 29749 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 (DEFAULT_TABLE table=test-table [id=5eb15641442e47fdabc12a350362e77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:47.131161 29749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b5c33e1fa44bfcbc1cd79b73559356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:47.131474 29966 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Bootstrap starting.
I20260927 16:30:47.133914 29966 tablet_bootstrap.cc:654] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:47.134634 29966 log.cc:824] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:47.135223 29967 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap starting.
I20260927 16:30:47.136657 29966 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: No bootstrap required, opened a new log
I20260927 16:30:47.136842 29966 ts_tablet_manager.cc:1397] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:30:47.137256 29967 tablet_bootstrap.cc:654] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:47.137758 29969 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Bootstrap starting.
I20260927 16:30:47.138373 29967 log.cc:824] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:47.139966 29967 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: No bootstrap required, opened a new log
I20260927 16:30:47.140182 29967 ts_tablet_manager.cc:1397] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:30:47.140232 29969 tablet_bootstrap.cc:654] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:47.140793 29966 raft_consensus.cc:348] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.140990 29969 log.cc:824] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:47.141078 29966 raft_consensus.cc:374] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:47.141151 29966 raft_consensus.cc:729] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a200477dd3ff4919ad3d81c41587cfd3, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.141635 29966 consensus_queue.cc:260] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.142319 29949 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
I20260927 16:30:47.142608 29966 ts_tablet_manager.cc:1428] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:30:47.142863 29969 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: No bootstrap required, opened a new log
I20260927 16:30:47.142990 29969 ts_tablet_manager.cc:1397] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:30:47.143882 29967 raft_consensus.cc:348] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.144145 29967 raft_consensus.cc:374] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:47.144207 29967 raft_consensus.cc:729] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.144649 29967 consensus_queue.cc:260] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.145241 29681 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
I20260927 16:30:47.145562 29967 ts_tablet_manager.cc:1428] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 16:30:47.147646 29969 raft_consensus.cc:348] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.147872 29969 raft_consensus.cc:374] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:47.147940 29969 raft_consensus.cc:729] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03be8534a2de44c7b82726b162e4bf35, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.148334 29969 consensus_queue.cc:260] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.148931 29815 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
I20260927 16:30:47.149231 29969 ts_tablet_manager.cc:1428] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:30:47.213049 29973 raft_consensus.cc:482] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:47.213300 29973 raft_consensus.cc:504] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.214412 29973 leader_election.cc:288] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:47.218288 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.214659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.15.93.129:34595)
0927 16:30:47.215084 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.215103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.215186 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.215485 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.215487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.215518 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:47.215654 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.215659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.216388 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.216390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.217219 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.217227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.217857 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.217963 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.217987 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.218046 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.218128 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":36,"spinlock_wait_cycles":5120,"thread_start_us":107,"threads_started":1}
I20260927 16:30:47.218287 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.214798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:34595 (local address 127.15.93.130:45707)
0927 16:30:47.214932 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.214937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.215177 (+ 240us) server_negotiation.cc:407] Connection header received
0927 16:30:47.215321 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.215323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.215365 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.215461 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:47.215785 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.216254 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217344 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.217741 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217866 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.217919 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.218085 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.218166 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:30:47.218331 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.214698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:52157 (local address 127.15.93.131:43651)
0927 16:30:47.214821 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.214826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.215075 (+ 249us) server_negotiation.cc:407] Connection header received
0927 16:30:47.215280 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.215286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.215343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.215462 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:47.215742 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.216249 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217374 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.217755 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.217931 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.218159 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.218225 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:30:47.218338 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.214639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43651 (local address 127.15.93.129:52157)
0927 16:30:47.214974 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.214993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.215131 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.215481 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.215484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.215510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:47.215635 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.215640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.216366 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.216372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.217230 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.217239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.217894 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.217898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.218012 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.218054 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.218139 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.218201 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":46,"spinlock_wait_cycles":2304,"thread_start_us":80,"threads_started":1}
I20260927 16:30:47.219071 29903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5c33e1fa44bfcbc1cd79b73559356" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a200477dd3ff4919ad3d81c41587cfd3" is_pre_election: true
I20260927 16:30:47.219074 29769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5c33e1fa44bfcbc1cd79b73559356" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03be8534a2de44c7b82726b162e4bf35" is_pre_election: true
I20260927 16:30:47.219363 29903 raft_consensus.cc:2454] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 0.
I20260927 16:30:47.219350 29769 raft_consensus.cc:2454] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 0.
I20260927 16:30:47.219832 29570 leader_election.cc:302] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03be8534a2de44c7b82726b162e4bf35, 4745e9da5a6547ef90f2c86756c71e92; no voters:
I20260927 16:30:47.220109 29973 raft_consensus.cc:2781] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:30:47.220182 29973 raft_consensus.cc:482] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:30:47.220213 29973 raft_consensus.cc:3037] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:47.221220 29973 raft_consensus.cc:504] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.221589 29973 leader_election.cc:288] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 election: Requested vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:47.221886 29903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5c33e1fa44bfcbc1cd79b73559356" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a200477dd3ff4919ad3d81c41587cfd3"
I20260927 16:30:47.221952 29769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5c33e1fa44bfcbc1cd79b73559356" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03be8534a2de44c7b82726b162e4bf35"
I20260927 16:30:47.222038 29903 raft_consensus.cc:3037] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:47.222064 29769 raft_consensus.cc:3037] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:47.223038 29769 raft_consensus.cc:2454] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 1.
I20260927 16:30:47.223049 29903 raft_consensus.cc:2454] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 1.
I20260927 16:30:47.223299 29570 leader_election.cc:302] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03be8534a2de44c7b82726b162e4bf35, 4745e9da5a6547ef90f2c86756c71e92; no voters:
I20260927 16:30:47.223491 29973 raft_consensus.cc:2781] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:47.223737 29973 raft_consensus.cc:686] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 LEADER]: Becoming Leader. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Running, Role: LEADER
I20260927 16:30:47.224052 29973 consensus_queue.cc:237] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.227075 29489 catalog_manager.cc:5495] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "4745e9da5a6547ef90f2c86756c71e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } health_report { overall_health: UNKNOWN } } }
I20260927 16:30:47.242307 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.238775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.15.93.129:39469)
0927 16:30:47.238915 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.238919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.238934 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:30:47.239116 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.239119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.239173 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.239282 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:30:47.239556 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.240060 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.240904 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.241120 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.241225 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.241530 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:30:47.241936 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:30:47.242068 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.242109 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.242167 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:30:47.247869 29769 raft_consensus.cc:1261] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Refusing update from remote peer 4745e9da5a6547ef90f2c86756c71e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:47.247869 29903 raft_consensus.cc:1261] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Refusing update from remote peer 4745e9da5a6547ef90f2c86756c71e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:47.248569 29977 consensus_queue.cc:1035] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:47.248735 29973 consensus_queue.cc:1035] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:47.255424 29980 mvcc.cc:205] Tried to move back new op lower bound from 7333997147120631808 to 7333997147032465408. Current Snapshot: MvccSnapshot[applied={T|T < 7333997147120631808}]
I20260927 16:30:47.258436 29982 mvcc.cc:205] Tried to move back new op lower bound from 7333997147120631808 to 7333997147032465408. Current Snapshot: MvccSnapshot[applied={T|T < 7333997147120631808 or (T in {7333997147120631808})}]
I20260927 16:30:47.259173 29981 mvcc.cc:205] Tried to move back new op lower bound from 7333997147120631808 to 7333997147032465408. Current Snapshot: MvccSnapshot[applied={T|T < 7333997147120631808 or (T in {7333997147120631808})}]
I20260927 16:30:47.263616 29489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:47.263752 29489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:47.266086 29489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Sending DeleteTablet for 3 replicas of tablet f0b5c33e1fa44bfcbc1cd79b73559356
I20260927 16:30:47.266822 29883 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:47 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:34504
I20260927 16:30:47.266824 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:47 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:44114
I20260927 16:30:47.267055 29749 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:47 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48480
I20260927 16:30:47.267416 29990 tablet_replica.cc:330] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: stopping tablet replica
I20260927 16:30:47.267527 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29552
I20260927 16:30:47.271608 29991 tablet_replica.cc:330] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: stopping tablet replica
I20260927 16:30:47.271914 29991 raft_consensus.cc:2229] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:47.271946 29992 tablet_replica.cc:330] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: stopping tablet replica
I20260927 16:30:47.272243 29992 raft_consensus.cc:2229] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:47.272253 29991 raft_consensus.cc:2258] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:47.272581 29992 raft_consensus.cc:2258] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 16:30:47.273204 29475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:39469 (error 108)
I20260927 16:30:47.273332 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29686
I20260927 16:30:47.274580 29991 ts_tablet_manager.cc:1905] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:47.276309 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.275977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:39469 (local address 127.0.0.1:44132)
0927 16:30:47.276142 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.276218 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:39469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 16:30:47.276548 29475 catalog_manager.cc:4584] TS 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129:39469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0b5c33e1fa44bfcbc1cd79b73559356: Network error: Client connection negotiation failed: client connection to 127.15.93.129:39469: connect: Connection refused (error 111)
I20260927 16:30:47.277102 29991 ts_tablet_manager.cc:1918] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:47.277212 29991 log.cc:1196] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/f0b5c33e1fa44bfcbc1cd79b73559356
I20260927 16:30:47.277519 29991 ts_tablet_manager.cc:1939] T f0b5c33e1fa44bfcbc1cd79b73559356 P a200477dd3ff4919ad3d81c41587cfd3: Deleting consensus metadata
I20260927 16:30:47.278561 29477 catalog_manager.cc:4841] TS a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): tablet f0b5c33e1fa44bfcbc1cd79b73559356 (table test-table [id=5eb15641442e47fdabc12a350362e77a]) successfully deleted
I20260927 16:30:47.279659 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29820
I20260927 16:30:47.279968 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.279713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.0.0.1:48496)
0927 16:30:47.279826 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.279887 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.130:45707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 16:30:47.285207 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29458
I20260927 16:30:47.292932 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--webserver_interface=127.15.93.190
--webserver_port=42069
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:47.404764 29994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:47.405017 29994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:47.405057 29994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:47.409160 29994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:47.409212 29994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:47.409232 29994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:47.409283 29994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:47.409315 29994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:47.414656 29994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=42069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:47.415689 29994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:47.416664 29994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:47.429129 30000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.429129 30001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.429152 30003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:47.429518 29994 server_base.cc:1034] running on GCE node
I20260927 16:30:47.430137 29994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:47.431223 29994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:47.432379 29994 hybrid_clock.cc:648] HybridClock initialized: now 1790526647432365 us; error 30 us; skew 500 ppm
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:47.434995 29994 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:47.435837 29994 webserver.cc:460] Webserver started at http://127.15.93.190:42069/ using document root <none> and password file <none>
I20260927 16:30:47.436376 29994 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:47.436444 29994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:47.440136 29994 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:30:47.442101 30010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.443260 29994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:47.443411 29994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "a5cf9d69c1354c47a1dba03bfc61f63d"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "7f4e292f891c2f2a0f81fec6d9c9eb65"
server_key_iv: "4ead027593a30bdaf09a8d8ab30e8ef2"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.443974 29994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:47.453383 29994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:47.455998 29994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:47.456192 29994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:47.462868 29994 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:33681
I20260927 16:30:47.463135 30062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:33681 every 8 connection(s)
I20260927 16:30:47.463979 29994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:47.468217 30063 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:30:47.468875 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 29994
I20260927 16:30:47.469293 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:39469
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=44533
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:47.469888 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap starting.
I20260927 16:30:47.487617 30063 log.cc:824] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:47.496068 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:47.496423 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap complete.
I20260927 16:30:47.500412 30063 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:47.500780 30063 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.501235 30063 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:47.501372 30063 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:47.501462 30063 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:30:47.501574 30063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:30:47.503355 30063 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:47.503733 30063 leader_election.cc:302] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5cf9d69c1354c47a1dba03bfc61f63d; no voters:
I20260927 16:30:47.504046 30063 leader_election.cc:288] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:30:47.504181 30067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:30:47.504418 30067 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 LEADER]: Becoming Leader. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Running, Role: LEADER
I20260927 16:30:47.504715 30067 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:47.505182 30063 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:47.505616 30069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5cf9d69c1354c47a1dba03bfc61f63d. Latest consensus state: current_term: 2 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:47.505730 30069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:47.505982 30068 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:47.506068 30071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:47.506110 30068 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:47.510576 30071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:47.511360 30071 catalog_manager.cc:1259] Loaded cluster ID: c17869e283d04e12989f160abcc100df
I20260927 16:30:47.511438 30071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:30:47.512814 30071 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 16:30:47.517294 30071 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 16:30:47.522501 30071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:30:47.523208 30071 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Loaded TSK: 0
I20260927 16:30:47.524358 30071 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:30:47.587172 30065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:47.587493 30065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:47.587540 30065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:47.591622 30065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:47.591696 30065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:47.591797 30065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:30:47.597100 30065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=44533
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:47.598042 30065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:47.599007 30065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:47.609364 30093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.609354 30091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.609354 30090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:47.610169 30065 server_base.cc:1034] running on GCE node
I20260927 16:30:47.610543 30065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:47.611142 30065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:47.612320 30065 hybrid_clock.cc:648] HybridClock initialized: now 1790526647612280 us; error 58 us; skew 500 ppm
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:47.615068 30065 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:30:47.616025 30065 webserver.cc:460] Webserver started at http://127.15.93.129:44533/ using document root <none> and password file <none>
I20260927 16:30:47.616572 30065 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:47.616673 30065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:47.620695 30065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:30:47.622828 30100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.624063 30065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:30:47.624155 30065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "4745e9da5a6547ef90f2c86756c71e92"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "77554f6c6cc408a1f7423236845a6c73"
server_key_iv: "695b2285db1562346137f63a6e926a41"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.624691 30065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:47.653056 30065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:47.655918 30065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:47.656117 30065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:47.656694 30065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:47.657704 30107 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:30:47.661677 30065 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:30:47.661731 30065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 16:30:47.661778 30065 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:30:47.663455 30065 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:30:47.663501 30065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260927 16:30:47.663781 30107 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap starting.
I20260927 16:30:47.673759 30065 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:39469
I20260927 16:30:47.673797 30214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:39469 every 8 connection(s)
I20260927 16:30:47.674687 30065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:30:47.679706 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30065
I20260927 16:30:47.680178 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:45707
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=45537
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:47.683046 30107 log.cc:824] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:47.690675 30107 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:47.691116 30107 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap complete.
I20260927 16:30:47.692454 30107 ts_tablet_manager.cc:1397] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.020s
I20260927 16:30:47.694362 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.676873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:49725 (local address 127.15.93.190:33681)
0927 16:30:47.677348 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.677366 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.677414 (+ 48us) server_negotiation.cc:407] Connection header received
0927 16:30:47.678550 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.678565 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.678974 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.679439 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.680877 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.682377 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.684231 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.684899 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.685074 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.688356 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.688364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.688382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.688423 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.690759 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.691416 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.691425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.691435 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.691599 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.692118 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.692121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.692123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.692548 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.692680 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.693272 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.693563 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":133,"threads_started":1}
I20260927 16:30:47.694757 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.676794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:49725)
0927 16:30:47.677265 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.677316 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.678115 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.679486 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.679495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.679984 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:47.680663 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.680686 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.682551 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.682557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.684068 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.684083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.685071 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.685078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.685324 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.686064 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:47.686098 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:47.688136 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:47.690872 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.690878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.690897 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.691254 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.691708 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.691714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.691719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.691895 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.692670 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:47.692678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:47.692956 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.693140 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.693444 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260927 16:30:47.695348 30107 raft_consensus.cc:348] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.695873 30107 raft_consensus.cc:729] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.696369 30107 consensus_queue.cc:260] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.697309 30215 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:47.697358 30107 ts_tablet_manager.cc:1428] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 16:30:47.697535 30215 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:47.698020 30215 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:47.700404 30027 ts_manager.cc:194] Registered new tserver with Master: 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129:39469)
I20260927 16:30:47.703596 30027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:49725
I20260927 16:30:47.705941 30215 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:30:47.713367 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.703097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.15.93.129:39469)
0927 16:30:47.703465 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.703476 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.703843 (+ 367us) server_negotiation.cc:407] Connection header received
0927 16:30:47.704038 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.704043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.704232 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.704356 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.705181 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.706172 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.707150 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.707638 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.707738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.710000 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.710008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.710018 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.710038 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.711651 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.712160 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.712164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.712165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.712237 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.712502 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.712504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.712505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.712701 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.712806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.713133 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.713251 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":93,"threads_started":1}
I20260927 16:30:47.713402 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.703020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:39469 (local address 127.0.0.1:44146)
0927 16:30:47.703719 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.703761 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.703892 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.704502 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.704508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.704921 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:47.705040 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.705048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.706321 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.706323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.707027 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.707040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.707752 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.707755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.707864 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.708363 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:47.708377 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:47.709867 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:47.711754 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.711763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.711768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.712045 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.712323 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.712326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.712328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.712418 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.712812 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:47.712818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:47.712929 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.713067 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.713245 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":118,"threads_started":1}
I20260927 16:30:47.714344 30149 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:47 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:44146
I20260927 16:30:47.715282 30224 tablet_replica.cc:330] stopping tablet replica
I20260927 16:30:47.715610 30224 raft_consensus.cc:2229] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:47.715847 30224 raft_consensus.cc:2258] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:47.716743 30224 ts_tablet_manager.cc:1905] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:47.720047 30224 ts_tablet_manager.cc:1918] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:47.720160 30224 log.cc:1196] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/f0b5c33e1fa44bfcbc1cd79b73559356
I20260927 16:30:47.720419 30224 ts_tablet_manager.cc:1939] T f0b5c33e1fa44bfcbc1cd79b73559356 P 4745e9da5a6547ef90f2c86756c71e92: Deleting consensus metadata
I20260927 16:30:47.721944 30012 catalog_manager.cc:4841] TS 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129:39469): tablet f0b5c33e1fa44bfcbc1cd79b73559356 (table test-table [id=5eb15641442e47fdabc12a350362e77a]) successfully deleted
W20260927 16:30:47.797439 30219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:47.797689 30219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:47.797732 30219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:47.801563 30219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:47.801614 30219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:47.801669 30219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:30:47.806996 30219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:45707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=45537
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:47.808018 30219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:47.808962 30219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:47.818140 30232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.818260 30229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:47.818149 30230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:47.818454 30219 server_base.cc:1034] running on GCE node
I20260927 16:30:47.818970 30219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:47.819641 30219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:47.820825 30219 hybrid_clock.cc:648] HybridClock initialized: now 1790526647820809 us; error 36 us; skew 500 ppm
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:47.823630 30219 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:30:47.824519 30219 webserver.cc:460] Webserver started at http://127.15.93.130:45537/ using document root <none> and password file <none>
I20260927 16:30:47.825089 30219 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:47.825202 30219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:47.829166 30219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:30:47.831229 30239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:47.832469 30219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:47.832613 30219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "03be8534a2de44c7b82726b162e4bf35"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "1cb688f39a4358c7a853170ac15077fe"
server_key_iv: "e7e68fab44596d112b6d3a71d3a3f29f"
server_key_version: "encryptionkey@0"
I20260927 16:30:47.833168 30219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:47.843912 30219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:47.846601 30219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:47.846799 30219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:47.847479 30219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:47.848577 30246 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:30:47.852500 30219 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:30:47.852584 30219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 16:30:47.852659 30219 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:30:47.854225 30219 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:30:47.854293 30219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:30:47.854528 30246 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Bootstrap starting.
I20260927 16:30:47.864509 30219 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:45707
I20260927 16:30:47.864559 30353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:45707 every 8 connection(s)
I20260927 16:30:47.865411 30219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:30:47.866916 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30219
I20260927 16:30:47.867398 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:43651
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=37449
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:47.876070 30246 log.cc:824] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:47.883013 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.867667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:34379)
0927 16:30:47.868268 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.868324 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.869514 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.870163 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.870172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.870701 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:47.871442 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.871462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.872270 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.872273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.873780 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.873794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.874281 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.874287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.874550 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.875279 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:47.875306 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:47.877261 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:47.879867 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.879875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.879893 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.880278 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.880650 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.880653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.880656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.880830 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.881291 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:47.881297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:47.881567 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.881757 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.882068 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":152,"threads_started":1}
I20260927 16:30:47.883263 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.867477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:34379 (local address 127.15.93.190:33681)
0927 16:30:47.867627 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.867635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.868427 (+ 792us) server_negotiation.cc:407] Connection header received
0927 16:30:47.869808 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.869812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.869880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.869984 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.871594 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.872124 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.873903 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.874138 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.874216 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.877444 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.877447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.877449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.877473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.879722 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.880412 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.880418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.880422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.880521 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.880935 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.880939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.880940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.881157 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.881242 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.882945 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.883077 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:30:47.884272 30246 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:47.884546 30354 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:47.884678 30246 tablet_bootstrap.cc:492] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Bootstrap complete.
I20260927 16:30:47.884781 30354 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:47.885339 30354 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:47.885813 30246 ts_tablet_manager.cc:1397] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20260927 16:30:47.887097 30027 ts_manager.cc:194] Registered new tserver with Master: 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:47.888447 30027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:34379
I20260927 16:30:47.888254 30246 raft_consensus.cc:348] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.888784 30246 raft_consensus.cc:729] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03be8534a2de44c7b82726b162e4bf35, State: Initialized, Role: FOLLOWER
I20260927 16:30:47.889284 30246 consensus_queue.cc:260] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:47.890311 30246 ts_tablet_manager.cc:1428] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 16:30:47.890684 30354 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
Sep 27 16:30:47 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.130@KRBTEST.COM
I20260927 16:30:47.898507 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.888572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.15.93.130:45707)
0927 16:30:47.889048 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:47.889060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:47.889081 (+ 21us) server_negotiation.cc:407] Connection header received
0927 16:30:47.889140 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:47.889145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:47.889332 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:47.889988 (+ 656us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:47.890095 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.891158 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.892392 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.892864 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.892982 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.895225 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:47.895233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:47.895243 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:47.895263 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:47.896904 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.897362 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.897366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.897368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.897440 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:47.897707 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:47.897709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:47.897710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:47.897919 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:47.898017 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:47.898271 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 16:30:47.898383 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":110,"threads_started":1}
I20260927 16:30:47.898517 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:47.888083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.0.0.1:48500)
0927 16:30:47.888342 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:47.888357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:47.888433 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:47.889549 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:47.889555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:47.889748 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:47.889845 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.889852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.891312 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.891315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.892285 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:47.892296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:47.893094 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:47.893104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:47.893177 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:47.893646 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:47.893660 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:47.895086 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:47.897016 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.897019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.897021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.897257 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.897523 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:47.897526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:47.897528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:47.897624 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:47.898012 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:47.898014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:47.898110 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 16:30:47.898219 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 16:30:47.898362 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260927 16:30:47.899564 30288 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5c33e1fa44bfcbc1cd79b73559356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:47 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48500
I20260927 16:30:47.900487 30361 tablet_replica.cc:330] stopping tablet replica
I20260927 16:30:47.900792 30361 raft_consensus.cc:2229] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:47.901033 30361 raft_consensus.cc:2258] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:47.901968 30361 ts_tablet_manager.cc:1905] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:47.905535 30361 ts_tablet_manager.cc:1918] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:47.905619 30361 log.cc:1196] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/f0b5c33e1fa44bfcbc1cd79b73559356
I20260927 16:30:47.905918 30361 ts_tablet_manager.cc:1939] T f0b5c33e1fa44bfcbc1cd79b73559356 P 03be8534a2de44c7b82726b162e4bf35: Deleting consensus metadata
I20260927 16:30:47.907545 30013 catalog_manager.cc:4841] TS 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): tablet f0b5c33e1fa44bfcbc1cd79b73559356 (table test-table [id=5eb15641442e47fdabc12a350362e77a]) successfully deleted
W20260927 16:30:47.987411 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:47.987709 30357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:47.987792 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:47.991833 30357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:47.991909 30357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:47.991999 30357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:30:47.997910 30357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=37449
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:47.998980 30357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:48.000025 30357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:48.009491 30369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.009529 30367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.009549 30366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:48.009837 30357 server_base.cc:1034] running on GCE node
I20260927 16:30:48.010198 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:48.010794 30357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:48.011987 30357 hybrid_clock.cc:648] HybridClock initialized: now 1790526648011957 us; error 41 us; skew 500 ppm
Sep 27 16:30:48 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:48.014750 30357 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:30:48.015693 30357 webserver.cc:460] Webserver started at http://127.15.93.131:37449/ using document root <none> and password file <none>
I20260927 16:30:48.016193 30357 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:48.016259 30357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:48.019910 30357 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:30:48.021817 30376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.022995 30357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:48.023113 30357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "a200477dd3ff4919ad3d81c41587cfd3"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "b33d9a80a6615f9526a43bd4b061463c"
server_key_iv: "ff4b3fa442363e117561ec35c79f577d"
server_key_version: "encryptionkey@0"
I20260927 16:30:48.023697 30357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:48.048727 30357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:48.051800 30357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:48.052002 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:48.052595 30357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:48.053412 30357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:48.053460 30357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.053529 30357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:48.053565 30357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.062193 30357 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:43651
I20260927 16:30:48.062220 30489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:43651 every 8 connection(s)
I20260927 16:30:48.063150 30357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:30:48.065740 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30357
Sep 27 16:30:48 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:30:48.074337 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.066360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.15.93.190:33681)
0927 16:30:48.066636 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.066641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.066797 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:30:48.066968 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.066973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.067031 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.067130 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:48.067927 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.068466 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.069314 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.069554 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.069665 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.071052 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:48.071058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:48.071065 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:48.071089 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:48.072650 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:48.073094 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:48.073099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:48.073101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:48.073212 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:48.073508 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:48.073512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:48.073513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:48.073682 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:48.073909 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.074039 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.074154 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260927 16:30:48.077427 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.064943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:38069 (local address 127.15.93.190:33681)
0927 16:30:48.065122 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.065126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.066271 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 16:30:48.067885 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.067888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.067934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.068014 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:48.069141 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.069646 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.070959 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.071181 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.071293 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.073665 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:48.073668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:48.073673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:48.073700 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:48.075467 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:48.076045 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:48.076048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:48.076050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:48.076157 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:48.076535 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:48.076538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:48.076539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:48.076689 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:48.076766 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.077231 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.077323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:30:48.078303 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.065330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:38069)
0927 16:30:48.066108 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.066168 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:48.067684 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:48.068193 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:48.068200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:48.068541 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:48.068998 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.069010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.069783 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.069787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.070826 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.070837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.071332 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.071337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.071543 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.072044 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:48.072059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:48.073505 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:48.075577 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:48.075583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:48.075599 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:48.075919 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:48.076254 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:48.076258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:48.076260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:48.076445 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:48.076779 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:48.076784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:48.077033 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 16:30:48.077189 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 16:30:48.077438 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":101,"threads_started":1}
I20260927 16:30:48.079360 30490 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33681
I20260927 16:30:48.079623 30490 heartbeater.cc:464] Registering TS with master...
I20260927 16:30:48.080056 30490 heartbeater.cc:511] Master 127.15.93.190:33681 requested a full tablet report, sending...
I20260927 16:30:48.081004 30027 ts_manager.cc:194] Registered new tserver with Master: a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651)
I20260927 16:30:48.081525 30027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:38069
I20260927 16:30:48.088023 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:30:48.092145 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.088597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.15.93.190:33681)
0927 16:30:48.088718 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.088722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.088924 (+ 202us) server_negotiation.cc:407] Connection header received
0927 16:30:48.089063 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.089065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.089162 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.089237 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:30:48.089439 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.089926 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.090760 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.090964 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.091044 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.091285 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:30:48.091711 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:30:48.091800 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.091982 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.092041 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:30:48.092698 30027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:30:48.095973 30027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:30:48.102473 30149 tablet_service.cc:1467] Processing CreateTablet for tablet f372c24d07fc4c098903e5fb993706f4 (DEFAULT_TABLE table=test-table [id=9800871ef6a64fb9b30b9bcf9eed4c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:48.102471 30288 tablet_service.cc:1467] Processing CreateTablet for tablet f372c24d07fc4c098903e5fb993706f4 (DEFAULT_TABLE table=test-table [id=9800871ef6a64fb9b30b9bcf9eed4c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:48.103061 30288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f372c24d07fc4c098903e5fb993706f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:48.103061 30149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f372c24d07fc4c098903e5fb993706f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:30:48.105832 30246 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Bootstrap starting.
I20260927 16:30:48.105841 30107 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap starting.
I20260927 16:30:48.106729 30107 tablet_bootstrap.cc:654] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:48.106756 30246 tablet_bootstrap.cc:654] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:48.106778 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.101804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43651 (local address 127.0.0.1:34514)
0927 16:30:48.101954 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.101969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:48.102040 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:48.103001 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:48.103004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:48.103021 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:48.103118 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.103124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.104186 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.104192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.105030 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.105040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.106318 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.106322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.106412 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.106456 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 16:30:48.106534 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:30:48.106614 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 16:30:48.107234 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.102044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.15.93.131:43651)
0927 16:30:48.102524 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.102536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.102561 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:30:48.102641 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.102651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.102853 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.103025 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:48.103231 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.104042 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.105160 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.106192 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.106309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.106443 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.106946 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.107073 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":176,"threads_started":1}
I20260927 16:30:48.108215 30246 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: No bootstrap required, opened a new log
I20260927 16:30:48.108315 30246 ts_tablet_manager.cc:1397] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:30:48.108794 30246 raft_consensus.cc:348] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.108886 30107 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: No bootstrap required, opened a new log
I20260927 16:30:48.108913 30246 raft_consensus.cc:374] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:48.108955 30107 ts_tablet_manager.cc:1397] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:30:48.108951 30246 raft_consensus.cc:729] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03be8534a2de44c7b82726b162e4bf35, State: Initialized, Role: FOLLOWER
I20260927 16:30:48.109086 30246 consensus_queue.cc:260] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.109495 30246 ts_tablet_manager.cc:1428] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:30:48.109416 30107 raft_consensus.cc:348] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.109539 30107 raft_consensus.cc:374] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:48.109575 30107 raft_consensus.cc:729] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Initialized, Role: FOLLOWER
I20260927 16:30:48.109697 30107 consensus_queue.cc:260] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.109959 30107 ts_tablet_manager.cc:1428] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:30:48.111191 30424 tablet_service.cc:1467] Processing CreateTablet for tablet f372c24d07fc4c098903e5fb993706f4 (DEFAULT_TABLE table=test-table [id=9800871ef6a64fb9b30b9bcf9eed4c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:30:48.112769 30424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f372c24d07fc4c098903e5fb993706f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 16:30:48.116377 30355 tablet.cc:2369] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:30:48.118897 30502 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Bootstrap starting.
I20260927 16:30:48.121428 30502 tablet_bootstrap.cc:654] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Neither blocks nor log segments found. Creating new log.
I20260927 16:30:48.122121 30502 log.cc:824] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:48.123812 30502 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: No bootstrap required, opened a new log
I20260927 16:30:48.124001 30502 ts_tablet_manager.cc:1397] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:30:48.126982 30502 raft_consensus.cc:348] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.127255 30502 raft_consensus.cc:374] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:30:48.127314 30502 raft_consensus.cc:729] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a200477dd3ff4919ad3d81c41587cfd3, State: Initialized, Role: FOLLOWER
I20260927 16:30:48.127764 30502 consensus_queue.cc:260] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.128374 30490 heartbeater.cc:503] Master 127.15.93.190:33681 was elected leader, sending a full tablet report...
I20260927 16:30:48.128690 30502 ts_tablet_manager.cc:1428] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 16:30:48.175853 30216 tablet.cc:2369] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:30:48.179602 30221 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:48.179838 30221 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.180980 30221 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:48.185803 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.181320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:56841 (local address 127.15.93.130:45707)
0927 16:30:48.181447 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.181451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.181719 (+ 268us) server_negotiation.cc:407] Connection header received
0927 16:30:48.181809 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.181811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.181887 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.181960 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:48.182240 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.182957 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.184608 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.185286 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.185386 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.185506 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.185602 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.185673 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:30:48.185798 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.181406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:60467 (local address 127.15.93.131:43651)
0927 16:30:48.181537 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.181543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.181560 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:30:48.181622 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.181624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.181662 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.181764 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:30:48.182058 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.182848 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.184608 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.185192 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.185319 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.185382 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.185619 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.185689 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:30:48.185814 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.181216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43651 (local address 127.15.93.129:60467)
0927 16:30:48.181367 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.181386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:48.181489 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:48.181792 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:48.181798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:48.181816 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:48.181922 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.181926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.182954 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.182957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.184482 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.184493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.185307 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.185310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.185446 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.185498 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 16:30:48.185578 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 16:30:48.185652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":94}
I20260927 16:30:48.185830 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.181216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.15.93.129:56841)
0927 16:30:48.181593 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.181625 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:48.181710 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:48.182001 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:48.182006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:48.182043 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:30:48.182133 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.182137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.183059 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.183062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.184482 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:48.184493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.185394 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.185399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:48.185475 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.185499 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:30:48.185578 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:30:48.185651 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":254,"thread_start_us":99,"threads_started":1}
I20260927 16:30:48.186452 30444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f372c24d07fc4c098903e5fb993706f4" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a200477dd3ff4919ad3d81c41587cfd3" is_pre_election: true
I20260927 16:30:48.186741 30444 raft_consensus.cc:2454] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 0.
I20260927 16:30:48.186686 30307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f372c24d07fc4c098903e5fb993706f4" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03be8534a2de44c7b82726b162e4bf35" is_pre_election: true
I20260927 16:30:48.186975 30307 raft_consensus.cc:2454] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 0.
I20260927 16:30:48.187211 30104 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92, a200477dd3ff4919ad3d81c41587cfd3; no voters:
I20260927 16:30:48.187484 30221 raft_consensus.cc:2781] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:30:48.187556 30221 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:30:48.187611 30221 raft_consensus.cc:3037] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:48.188601 30221 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.188944 30221 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 election: Requested vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
I20260927 16:30:48.189292 30444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f372c24d07fc4c098903e5fb993706f4" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a200477dd3ff4919ad3d81c41587cfd3"
I20260927 16:30:48.189292 30307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f372c24d07fc4c098903e5fb993706f4" candidate_uuid: "4745e9da5a6547ef90f2c86756c71e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03be8534a2de44c7b82726b162e4bf35"
I20260927 16:30:48.189412 30307 raft_consensus.cc:3037] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:48.189412 30444 raft_consensus.cc:3037] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:30:48.190663 30444 raft_consensus.cc:2454] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 1.
I20260927 16:30:48.190663 30307 raft_consensus.cc:2454] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4745e9da5a6547ef90f2c86756c71e92 in term 1.
I20260927 16:30:48.190951 30103 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03be8534a2de44c7b82726b162e4bf35, 4745e9da5a6547ef90f2c86756c71e92; no voters:
I20260927 16:30:48.191149 30221 raft_consensus.cc:2781] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:30:48.191453 30221 raft_consensus.cc:686] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 LEADER]: Becoming Leader. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Running, Role: LEADER
I20260927 16:30:48.191736 30221 consensus_queue.cc:237] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.194722 30025 catalog_manager.cc:5495] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "4745e9da5a6547ef90f2c86756c71e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } health_report { overall_health: UNKNOWN } } }
I20260927 16:30:48.206550 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.203174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.15.93.129:39469)
0927 16:30:48.203295 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:48.203298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:48.203311 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:30:48.203491 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:48.203494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:48.203546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:48.203636 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:30:48.203840 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.204358 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.205156 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:48.205377 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:48.205453 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:48.205811 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:30:48.206242 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:30:48.206322 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:48.206405 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 16:30:48.206453 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:30:48.210357 30307 raft_consensus.cc:1261] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Refusing update from remote peer 4745e9da5a6547ef90f2c86756c71e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:48.210361 30444 raft_consensus.cc:1261] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Refusing update from remote peer 4745e9da5a6547ef90f2c86756c71e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:30:48.211045 30506 consensus_queue.cc:1035] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:48.211189 30221 consensus_queue.cc:1035] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:30:48.216218 30508 mvcc.cc:205] Tried to move back new op lower bound from 7333997151063564288 to 7333997150995984384. Current Snapshot: MvccSnapshot[applied={T|T < 7333997151063564288}]
I20260927 16:30:48.225418 30026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:48.225566 30026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:30:48.227962 30026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Sending DeleteTablet for 3 replicas of tablet f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:48.228582 30149 tablet_service.cc:1514] Processing DeleteTablet for tablet f372c24d07fc4c098903e5fb993706f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:48 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:44146
I20260927 16:30:48.228673 30424 tablet_service.cc:1514] Processing DeleteTablet for tablet f372c24d07fc4c098903e5fb993706f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:48 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:34514
I20260927 16:30:48.228969 30518 tablet_replica.cc:330] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: stopping tablet replica
I20260927 16:30:48.229105 30518 raft_consensus.cc:2229] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:30:48.229095 30288 tablet_service.cc:1514] Processing DeleteTablet for tablet f372c24d07fc4c098903e5fb993706f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:30:48 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48500
I20260927 16:30:48.229229 30519 tablet_replica.cc:330] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: stopping tablet replica
I20260927 16:30:48.229267 30361 tablet_replica.cc:330] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: stopping tablet replica
I20260927 16:30:48.229362 30361 raft_consensus.cc:2229] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:48.229382 30518 raft_consensus.cc:2258] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:48.229517 30361 raft_consensus.cc:2258] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:48.229645 30519 raft_consensus.cc:2229] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:30:48.230154 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30065
I20260927 16:30:48.230644 30361 ts_tablet_manager.cc:1905] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:48.232115 30519 raft_consensus.cc:2258] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:30:48.233023 30361 ts_tablet_manager.cc:1918] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:48.233116 30361 log.cc:1196] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:48.233403 30361 ts_tablet_manager.cc:1939] T f372c24d07fc4c098903e5fb993706f4 P 03be8534a2de44c7b82726b162e4bf35: Deleting consensus metadata
I20260927 16:30:48.233737 30519 ts_tablet_manager.cc:1905] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:30:48.234025 30013 catalog_manager.cc:4841] TS 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): tablet f372c24d07fc4c098903e5fb993706f4 (table test-table [id=9800871ef6a64fb9b30b9bcf9eed4c79]) successfully deleted
I20260927 16:30:48.237066 30519 ts_tablet_manager.cc:1918] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:30:48.237180 30519 log.cc:1196] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:48.237519 30519 ts_tablet_manager.cc:1939] T f372c24d07fc4c098903e5fb993706f4 P a200477dd3ff4919ad3d81c41587cfd3: Deleting consensus metadata
W20260927 16:30:48.238533 30012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:39469 (error 108)
I20260927 16:30:48.239030 30014 catalog_manager.cc:4841] TS a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): tablet f372c24d07fc4c098903e5fb993706f4 (table test-table [id=9800871ef6a64fb9b30b9bcf9eed4c79]) successfully deleted
I20260927 16:30:48.239450 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30219
I20260927 16:30:48.239755 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.239455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:39469 (local address 127.0.0.1:44174)
0927 16:30:48.239584 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.239661 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:39469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:30:48.239898 30012 catalog_manager.cc:4584] TS 4745e9da5a6547ef90f2c86756c71e92 (127.15.93.129:39469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f372c24d07fc4c098903e5fb993706f4: Network error: Client connection negotiation failed: client connection to 127.15.93.129:39469: connect: Connection refused (error 111)
I20260927 16:30:48.246209 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30357
I20260927 16:30:48.251485 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 29994
I20260927 16:30:48.257838 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--webserver_interface=127.15.93.190
--webserver_port=42069
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:30:48.367025 30521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:48.367283 30521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:48.367322 30521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:48.371322 30521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:30:48.371392 30521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:48.371411 30521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:48.371428 30521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:30:48.371444 30521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:30:48.376677 30521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=42069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:48.377625 30521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:48.378587 30521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:48.386765 30526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.386759 30529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.386759 30527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:48.386957 30521 server_base.cc:1034] running on GCE node
I20260927 16:30:48.387775 30521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:48.388923 30521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:48.390084 30521 hybrid_clock.cc:648] HybridClock initialized: now 1790526648390068 us; error 29 us; skew 500 ppm
Sep 27 16:30:48 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:48.392892 30521 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:30:48.393752 30521 webserver.cc:460] Webserver started at http://127.15.93.190:42069/ using document root <none> and password file <none>
I20260927 16:30:48.394302 30521 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:48.394361 30521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:48.398154 30521 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:30:48.400454 30536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.401593 30521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:30:48.401719 30521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "a5cf9d69c1354c47a1dba03bfc61f63d"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "7f4e292f891c2f2a0f81fec6d9c9eb65"
server_key_iv: "4ead027593a30bdaf09a8d8ab30e8ef2"
server_key_version: "encryptionkey@0"
I20260927 16:30:48.402254 30521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:48.433260 30521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:48.436201 30521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:48.436403 30521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:48.443234 30521 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:33681
I20260927 16:30:48.443567 30588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:33681 every 8 connection(s)
I20260927 16:30:48.444424 30521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:30:48.448645 30589 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:30:48.450284 30589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap starting.
I20260927 16:30:48.454005 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30521
I20260927 16:30:48.454474 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:39469
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=44533
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:48.467854 30589 log.cc:824] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:48.478950 30589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:48.479331 30589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d: Bootstrap complete.
I20260927 16:30:48.483237 30589 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:48.483703 30589 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Initialized, Role: FOLLOWER
I20260927 16:30:48.484216 30589 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:48.484361 30589 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:30:48.484448 30589 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:30:48.484562 30589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:30:48.486369 30589 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:48.486701 30589 leader_election.cc:302] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5cf9d69c1354c47a1dba03bfc61f63d; no voters:
I20260927 16:30:48.487035 30589 leader_election.cc:288] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:30:48.487118 30593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:30:48.487317 30593 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [term 3 LEADER]: Becoming Leader. State: Replica: a5cf9d69c1354c47a1dba03bfc61f63d, State: Running, Role: LEADER
I20260927 16:30:48.487615 30593 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } }
I20260927 16:30:48.488243 30589 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:30:48.488691 30593 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5cf9d69c1354c47a1dba03bfc61f63d. Latest consensus state: current_term: 3 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:48.488794 30593 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:48.489207 30594 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf9d69c1354c47a1dba03bfc61f63d" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33681 } } }
I20260927 16:30:48.489387 30594 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf9d69c1354c47a1dba03bfc61f63d [sys.catalog]: This master's current role is: LEADER
I20260927 16:30:48.489217 30597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:30:48.494558 30597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:30:48.495246 30597 catalog_manager.cc:1259] Loaded cluster ID: c17869e283d04e12989f160abcc100df
I20260927 16:30:48.495324 30597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 16:30:48.496165 30597 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fab1059f9ad google::LogMessage::Fail() at ??:0
@ 0x7fab105a288a google::LogMessage::SendToLog() at ??:0
@ 0x7fab1059f4de google::LogMessage::Flush() at ??:0
@ 0x7fab105a0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fab1aedefdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fab1aedfd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fab1aedc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fab1af20640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fab127f13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fab127f1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fab127f33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fab127e2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fab13e266db start_thread at ??:0
@ 0x7fab0f09d61f clone at ??:0
W20260927 16:30:48.594200 30591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:48.594460 30591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:48.594502 30591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:48.598505 30591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:48.598559 30591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:48.598646 30591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:30:48.603853 30591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=44533
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:48.604851 30591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:48.605823 30591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:48.615110 30630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.615147 30631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.615410 30633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:48.615334 30591 server_base.cc:1034] running on GCE node
I20260927 16:30:48.615833 30591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:48.616492 30591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:48.617709 30591 hybrid_clock.cc:648] HybridClock initialized: now 1790526648617683 us; error 45 us; skew 500 ppm
Sep 27 16:30:48 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:48.620615 30591 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:30:48.621595 30591 webserver.cc:460] Webserver started at http://127.15.93.129:44533/ using document root <none> and password file <none>
I20260927 16:30:48.622159 30591 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:48.622217 30591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:48.626075 30591 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:30:48.628242 30640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.629490 30591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:30:48.629578 30591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "4745e9da5a6547ef90f2c86756c71e92"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "77554f6c6cc408a1f7423236845a6c73"
server_key_iv: "695b2285db1562346137f63a6e926a41"
server_key_version: "encryptionkey@0"
I20260927 16:30:48.630127 30591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:48.650882 30591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:48.654309 30591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:48.654515 30591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:48.655161 30591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:48.656287 30647 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:30:48.661653 30591 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:30:48.661731 30591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 16:30:48.661825 30591 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:30:48.663467 30591 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:30:48.663529 30591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:48.663826 30647 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap starting.
I20260927 16:30:48.674170 30591 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:39469
I20260927 16:30:48.674235 30754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:39469 every 8 connection(s)
I20260927 16:30:48.675096 30591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:30:48.681775 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30591
I20260927 16:30:48.682194 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:45707
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=45537
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:48.685875 30647 log.cc:824] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Log is configured to *not* fsync() on all Append() calls
I20260927 16:30:48.695186 30647 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:30:48.695693 30647 tablet_bootstrap.cc:492] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Bootstrap complete.
I20260927 16:30:48.697113 30647 ts_tablet_manager.cc:1397] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.016s
I20260927 16:30:48.700387 30647 raft_consensus.cc:348] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.700939 30647 raft_consensus.cc:729] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4745e9da5a6547ef90f2c86756c71e92, State: Initialized, Role: FOLLOWER
I20260927 16:30:48.701479 30647 consensus_queue.cc:260] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:48.702584 30647 ts_tablet_manager.cc:1428] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260927 16:30:48.712173 30757 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:30:48.676986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:50671)
0927 16:30:48.677455 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.677516 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:48.678308 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:48.711260 (+ 32952us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
W20260927 16:30:48.712823 30755 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:33681: Client connection negotiation failed: client connection to 127.15.93.190:33681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:30:48.713562 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.713330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:45681)
0927 16:30:48.713451 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.713513 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 16:30:48.714296 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.714071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:35531)
0927 16:30:48.714201 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.714248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 16:30:48.714572 30755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:30:48.797753 30758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:48.798007 30758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:48.798079 30758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:48.802042 30758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:48.802098 30758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:48.802191 30758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:30:48.807288 30758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:45707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=45537
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:48.808313 30758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:48.809305 30758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:48.818421 30766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.818419 30768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:48.818418 30765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:48.818516 30758 server_base.cc:1034] running on GCE node
I20260927 16:30:48.819128 30758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:48.819761 30758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:48.820919 30758 hybrid_clock.cc:648] HybridClock initialized: now 1790526648820899 us; error 32 us; skew 500 ppm
Sep 27 16:30:48 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:48.823555 30758 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:30:48.824365 30758 webserver.cc:460] Webserver started at http://127.15.93.130:45537/ using document root <none> and password file <none>
I20260927 16:30:48.824924 30758 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:48.824985 30758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:48.828789 30758 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 16:30:48.830833 30775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.832115 30758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:48.832245 30758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "03be8534a2de44c7b82726b162e4bf35"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "1cb688f39a4358c7a853170ac15077fe"
server_key_iv: "e7e68fab44596d112b6d3a71d3a3f29f"
server_key_version: "encryptionkey@0"
I20260927 16:30:48.832787 30758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:48.851594 30758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:48.854904 30758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:48.855108 30758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:48.855770 30758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:48.856583 30758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:48.856659 30758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.856731 30758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:48.856772 30758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:48.865393 30758 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:45707
I20260927 16:30:48.865422 30888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:45707 every 8 connection(s)
I20260927 16:30:48.866357 30758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:30:48.870051 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.868476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:50941)
0927 16:30:48.868950 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.869381 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260927 16:30:48.870344 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30758
W20260927 16:30:48.870532 30889 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:33681: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
I20260927 16:30:48.870730 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:43651
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:33681
--webserver_port=37449
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:30:48.871130 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.870839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:41043)
0927 16:30:48.870976 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.871038 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 16:30:48.871855 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:48.871588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:51877)
0927 16:30:48.871718 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:48.871777 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 16:30:48.872054 30889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:30:48.980379 30892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:30:48.980665 30892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:30:48.980710 30892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:30:48.984850 30892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:30:48.984910 30892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:30:48.985002 30892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:30:48.990331 30892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=37449
--tserver_master_addrs=127.15.93.190:33681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:30:48.991458 30892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:30:48.992539 30892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:30:49.001962 30900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:30:49.002005 30897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:49.001976 30892 server_base.cc:1034] running on GCE node
W20260927 16:30:49.002039 30898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:30:49.002519 30892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:30:49.003157 30892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:30:49.004383 30892 hybrid_clock.cc:648] HybridClock initialized: now 1790526649004335 us; error 64 us; skew 500 ppm
Sep 27 16:30:49 dist-test-slave-grzf krb5kdc[29445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526649, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:30:49.007149 30892 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:30:49.008036 30892 webserver.cc:460] Webserver started at http://127.15.93.131:37449/ using document root <none> and password file <none>
I20260927 16:30:49.008602 30892 fs_manager.cc:359] Metadata directory not provided
I20260927 16:30:49.008697 30892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:30:49.012492 30892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260927 16:30:49.014566 30907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:49.015854 30892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:30:49.015985 30892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "a200477dd3ff4919ad3d81c41587cfd3"
format_stamp: "Formatted at 2026-09-27 16:30:46 on dist-test-slave-grzf"
server_key: "b33d9a80a6615f9526a43bd4b061463c"
server_key_iv: "ff4b3fa442363e117561ec35c79f577d"
server_key_version: "encryptionkey@0"
I20260927 16:30:49.016525 30892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:30:49.049945 30892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:30:49.053462 30892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:30:49.053673 30892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:30:49.054286 30892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:30:49.055130 30892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:30:49.055199 30892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:49.055270 30892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:30:49.055312 30892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:30:49.064234 30892 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:43651
I20260927 16:30:49.064256 31020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:43651 every 8 connection(s)
I20260927 16:30:49.065276 30892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:30:49.067350 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 30892
I20260927 16:30:49.069232 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:49.067303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:53429)
0927 16:30:49.067898 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:49.068408 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":151,"threads_started":1}
W20260927 16:30:49.069653 31021 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:33681 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:33681: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
I20260927 16:30:49.070233 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:49.070024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:56977)
0927 16:30:49.070130 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:49.070184 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 16:30:49.070806 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:49.070608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:41155)
0927 16:30:49.070707 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:49.070750 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 16:30:49.070969 31021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:30:49.715636 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:49.715160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:41647)
0927 16:30:49.715466 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:49.715526 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260927 16:30:49.872954 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:49.872583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:46127)
0927 16:30:49.872807 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:49.872860 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260927 16:30:50.072176 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.071688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:48349)
0927 16:30:50.071987 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:50.072056 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
I20260927 16:30:50.438531 31028 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:50.438747 31028 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:50.439975 31028 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
Sep 27 16:30:50 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.130@KRBTEST.COM
Sep 27 16:30:50 dist-test-slave-grzf krb5kdc[29445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.131@KRBTEST.COM
I20260927 16:30:50.454510 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.440295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:45707 (local address 127.15.93.129:35921)
0927 16:30:50.440822 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:50.440837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:50.440988 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:50.442468 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:50.442474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:50.443160 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:50.443810 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:50.443830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.445160 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.445164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:50.446201 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:50.446212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.446972 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.446975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:50.447246 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:50.447949 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:50.447967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:50.449639 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:50.451954 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:50.451962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:50.451977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:50.452249 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:50.452650 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:50.452655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:50.452799 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:50.453457 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:50.453464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:50.453738 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 16:30:50.453933 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 16:30:50.454299 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":243,"spinlock_wait_cycles":4992,"thread_start_us":75,"threads_started":1}
I20260927 16:30:50.454741 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.440295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43651 (local address 127.15.93.129:59259)
0927 16:30:50.440690 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:50.440707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:30:50.440823 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:30:50.442468 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:30:50.442474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:30:50.443162 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:30:50.443809 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:50.443830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.445172 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.445175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:50.446202 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:30:50.446212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.446953 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.446957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:30:50.447213 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:50.447998 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:30:50.448010 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:30:50.449958 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:30:50.452250 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:50.452255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:50.452455 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:50.452816 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:30:50.452821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:30:50.452907 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:30:50.453556 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:30:50.453559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:30:50.453736 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 16:30:50.453934 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 16:30:50.454569 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":517,"spinlock_wait_cycles":7296,"thread_start_us":79,"threads_started":1}
I20260927 16:30:50.454756 31032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.440553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:59259 (local address 127.15.93.131:43651)
0927 16:30:50.440963 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:50.440981 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:50.441013 (+ 32us) server_negotiation.cc:407] Connection header received
0927 16:30:50.441684 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:50.441690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:50.442051 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:50.442393 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:50.444002 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.444993 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.446348 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.446816 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.447007 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:50.450155 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:50.450160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:50.450172 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:50.450202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:50.452117 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452572 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:50.452578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:50.452583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:50.452707 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452999 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:50.453004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:50.453005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:50.453447 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:50.453586 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:50.454271 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 16:30:50.454567 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":80,"threads_started":1}
I20260927 16:30:50.455593 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.440420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:35921 (local address 127.15.93.130:45707)
0927 16:30:50.440781 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 16:30:50.440797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:30:50.440964 (+ 167us) server_negotiation.cc:407] Connection header received
0927 16:30:50.441686 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:30:50.441692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:30:50.442051 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:30:50.442394 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:30:50.443998 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.445009 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.446335 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:30:50.446842 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:30:50.447029 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:30:50.449896 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:30:50.449901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:30:50.449915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:30:50.449946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:30:50.451824 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452382 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:50.452387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:50.452392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:50.452514 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:30:50.452896 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:30:50.452900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:30:50.452902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:30:50.453339 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:30:50.453485 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 16:30:50.455021 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0927 16:30:50.455320 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":78,"threads_started":1}
W20260927 16:30:50.457123 30644 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
W20260927 16:30:50.457479 30643 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:50.457574 30643 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92; no voters: 03be8534a2de44c7b82726b162e4bf35, a200477dd3ff4919ad3d81c41587cfd3
I20260927 16:30:50.457813 31028 raft_consensus.cc:2726] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:30:50.716846 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.716512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:50909)
0927 16:30:50.716689 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:50.716763 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 16:30:50.874076 31033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:50.873649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:35427)
0927 16:30:50.873914 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:50.873970 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260927 16:30:51.073463 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:51.072951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:49247)
0927 16:30:51.073306 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:51.073377 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260927 16:30:51.717960 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:51.717574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:51541)
0927 16:30:51.717813 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:51.717868 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 16:30:51.875363 31036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:51.874874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:48385)
0927 16:30:51.875204 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:51.875266 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":136,"threads_started":1}
I20260927 16:30:52.074702 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:52.074231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:49871)
0927 16:30:52.074527 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:52.074588 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260927 16:30:52.167881 31038 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:52.168043 31038 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:52.168547 31038 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
W20260927 16:30:52.169385 30643 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
W20260927 16:30:52.169654 30644 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:52.169749 30644 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92; no voters: 03be8534a2de44c7b82726b162e4bf35, a200477dd3ff4919ad3d81c41587cfd3
I20260927 16:30:52.169904 31038 raft_consensus.cc:2726] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:30:52.719179 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:52.718674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:37727)
0927 16:30:52.719021 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:52.719074 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":192,"threads_started":1}
I20260927 16:30:52.876523 31040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:52.876084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:56815)
0927 16:30:52.876355 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:52.876412 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260927 16:30:53.076025 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:53.075591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:57615)
0927 16:30:53.075858 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:53.075923 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260927 16:30:53.720592 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:53.720042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:38791)
0927 16:30:53.720416 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:53.720498 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":76,"threads_started":1}
I20260927 16:30:53.877656 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:53.877277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:58577)
0927 16:30:53.877509 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:53.877565 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260927 16:30:54.077180 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:54.076768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:40647)
0927 16:30:54.077030 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:54.077085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260927 16:30:54.721662 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:54.721267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:44211)
0927 16:30:54.721504 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:54.721560 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260927 16:30:54.873837 31046 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:54.873979 31046 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:54.874415 31046 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
W20260927 16:30:54.875149 30643 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
W20260927 16:30:54.875233 30644 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:54.875319 30644 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92; no voters: 03be8534a2de44c7b82726b162e4bf35, a200477dd3ff4919ad3d81c41587cfd3
I20260927 16:30:54.875468 31046 raft_consensus.cc:2726] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:30:54.878789 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:54.878304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:37187)
0927 16:30:54.878632 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:54.878703 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":65,"threads_started":1}
I20260927 16:30:55.078411 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:55.078005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:41903)
0927 16:30:55.078260 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:55.078320 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260927 16:30:55.722734 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:55.722362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:47501)
0927 16:30:55.722589 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:55.722641 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260927 16:30:55.879938 31050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:55.879492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:36209)
0927 16:30:55.879797 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:55.879851 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":85,"threads_started":1}
I20260927 16:30:56.079658 31051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:56.079166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:47991)
0927 16:30:56.079502 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:56.079559 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":84,"threads_started":1}
I20260927 16:30:56.723898 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:56.723515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:42781)
0927 16:30:56.723749 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:56.723805 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260927 16:30:56.881083 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:56.880653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:42911)
0927 16:30:56.880924 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:56.880980 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260927 16:30:57.080786 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:57.080391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:40481)
0927 16:30:57.080643 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:57.080695 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260927 16:30:57.652900 31055 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:30:57.653054 31055 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:30:57.653496 31055 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
W20260927 16:30:57.654210 30644 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
W20260927 16:30:57.654351 30643 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
I20260927 16:30:57.654458 30643 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92; no voters: 03be8534a2de44c7b82726b162e4bf35, a200477dd3ff4919ad3d81c41587cfd3
I20260927 16:30:57.654597 31055 raft_consensus.cc:2726] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:30:57.724982 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:57.724604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:36535)
0927 16:30:57.724842 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:57.724893 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260927 16:30:57.882176 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:57.881786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:35591)
0927 16:30:57.882030 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:57.882080 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260927 16:30:58.082088 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:58.081657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:55117)
0927 16:30:58.081937 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:58.081998 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
I20260927 16:30:58.726159 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:58.725773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:46173)
0927 16:30:58.726007 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:58.726060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260927 16:30:58.883292 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:58.882873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:57699)
0927 16:30:58.883143 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:58.883200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260927 16:30:59.083413 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:59.082963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:53463)
0927 16:30:59.083226 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:59.083280 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260927 16:30:59.727272 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:59.726871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:44583)
0927 16:30:59.727125 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:59.727178 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260927 16:30:59.884377 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:30:59.883984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:52383)
0927 16:30:59.884220 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 16:30:59.884284 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260927 16:31:00.084489 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:00.084114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:54861)
0927 16:31:00.084351 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:00.084400 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20260927 16:31:00.171325 31065 raft_consensus.cc:482] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:00.171525 31065 raft_consensus.cc:504] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a200477dd3ff4919ad3d81c41587cfd3" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43651 } } peers { permanent_uuid: "4745e9da5a6547ef90f2c86756c71e92" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 39469 } } peers { permanent_uuid: "03be8534a2de44c7b82726b162e4bf35" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 45707 } }
I20260927 16:31:00.172013 31065 leader_election.cc:288] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651), 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707)
W20260927 16:31:00.172725 30644 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a200477dd3ff4919ad3d81c41587cfd3 (127.15.93.131:43651): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
W20260927 16:31:00.172842 30643 leader_election.cc:341] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03be8534a2de44c7b82726b162e4bf35 (127.15.93.130:45707): Not found: Tablet not found: f372c24d07fc4c098903e5fb993706f4
I20260927 16:31:00.172940 30643 leader_election.cc:302] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4745e9da5a6547ef90f2c86756c71e92; no voters: 03be8534a2de44c7b82726b162e4bf35, a200477dd3ff4919ad3d81c41587cfd3
I20260927 16:31:00.173092 31065 raft_consensus.cc:2726] T f372c24d07fc4c098903e5fb993706f4 P 4745e9da5a6547ef90f2c86756c71e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:00.728492 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:00.728031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:47755)
0927 16:31:00.728338 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:00.728398 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260927 16:31:00.885509 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:00.885113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:53379)
0927 16:31:00.885364 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:00.885420 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260927 16:31:01.085693 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:01.085258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:48769)
0927 16:31:01.085540 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:01.085600 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260927 16:31:01.729624 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:01.729220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:57313)
0927 16:31:01.729476 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:01.729530 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260927 16:31:01.886644 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:01.886251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:36589)
0927 16:31:01.886491 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:01.886549 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260927 16:31:02.086803 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:02.086404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:39561)
0927 16:31:02.086652 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:02.086710 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260927 16:31:02.730885 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:02.730440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:45375)
0927 16:31:02.730720 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:02.730789 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260927 16:31:02.887939 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:02.887479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:49175)
0927 16:31:02.887778 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:02.887843 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260927 16:31:03.088158 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:03.087677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.131:42647)
0927 16:31:03.087996 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:03.088060 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":134,"threads_started":1}
I20260927 16:31:03.732213 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:03.731809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.129:53871)
0927 16:31:03.732066 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:03.732121 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260927 16:31:03.889261 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:03.888852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33681 (local address 127.15.93.130:51507)
0927 16:31:03.889117 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:03.889166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260927 16:31:04.072329 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30591
I20260927 16:31:04.078066 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30758
I20260927 16:31:04.083211 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 30892
2026-09-27T16:31:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19094 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:31:04 dist-test-slave-grzf krb5kdc[31079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:31:04 dist-test-slave-grzf krb5kdc[31079](info): set up 2 sockets
Sep 27 16:31:04 dist-test-slave-grzf krb5kdc[31079](info): commencing operation
krb5kdc: starting...
W20260927 16:31:05.123615 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:31:05.242220 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:05.349539 31092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:05.349918 31092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:05.350010 31092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:05.355176 31092 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:31:05.355232 31092 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:31:05.355248 31092 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:31:05.355262 31092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:31:05.355306 31092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:05.355338 31092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:05.355391 31092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:31:05.355427 31092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:31:05.360741 31092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:05.361764 31092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:05.362775 31092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:05.371353 31100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:05.371359 31098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:05.371440 31092 server_base.cc:1034] running on GCE node
W20260927 16:31:05.371359 31097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:05.372089 31092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:05.373198 31092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:05.374377 31092 hybrid_clock.cc:648] HybridClock initialized: now 1790526665374362 us; error 35 us; skew 500 ppm
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:05.377023 31092 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:31:05.377913 31092 webserver.cc:460] Webserver started at http://127.15.93.190:38083/ using document root <none> and password file <none>
I20260927 16:31:05.378459 31092 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:05.378545 31092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:05.378764 31092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:31:05.380784 31092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "1db7917f4c934e85837edcb7e357b616"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "484bbffe1f907e5a9efb162f4d7342d2"
server_key_iv: "5308c679e9af3ebda9abd451c6df6bf7"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.381345 31092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "1db7917f4c934e85837edcb7e357b616"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "484bbffe1f907e5a9efb162f4d7342d2"
server_key_iv: "5308c679e9af3ebda9abd451c6df6bf7"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.384817 31092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 16:31:05.386974 31107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.388186 31092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:05.388312 31092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "1db7917f4c934e85837edcb7e357b616"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "484bbffe1f907e5a9efb162f4d7342d2"
server_key_iv: "5308c679e9af3ebda9abd451c6df6bf7"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.388410 31092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:05.415302 31092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:05.417979 31092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:05.418176 31092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:05.424928 31092 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38483
I20260927 16:31:05.425194 31159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38483 every 8 connection(s)
I20260927 16:31:05.425855 31092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:31:05.427839 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31092
I20260927 16:31:05.427970 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:31:05.428247 15734 external_mini_cluster.cc:1428] Setting key 626195d435ba5470b4d13c05675968f8
I20260927 16:31:05.428548 31160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:05.435017 31160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap starting.
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:05.437613 31160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:05.438549 31160 log.cc:824] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:05.440622 31160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: No bootstrap required, opened a new log
I20260927 16:31:05.442107 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:05.429464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.15.93.190:38483)
0927 16:31:05.429850 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:05.429871 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:05.429902 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:31:05.430520 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:05.430536 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:05.430843 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:05.431103 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:05.431991 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.432953 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.434115 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.434561 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.434678 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:05.437329 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:05.437334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:05.437348 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:05.437376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:05.439178 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.439643 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.439649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.439654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.439780 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.440036 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.440039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.440041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.440499 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:05.440698 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:05.440956 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 16:31:05.441228 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
I20260927 16:31:05.444653 31160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:05.444847 31160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:05.444880 31160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Initialized, Role: FOLLOWER
I20260927 16:31:05.445214 31160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:05.445317 31160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:31:05.445358 31160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:31:05.445422 31160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:05.446171 31160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:05.446436 31160 leader_election.cc:302] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db7917f4c934e85837edcb7e357b616; no voters:
I20260927 16:31:05.446660 31160 leader_election.cc:288] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:31:05.446777 31165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:31:05.446995 31165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 LEADER]: Becoming Leader. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Running, Role: LEADER
I20260927 16:31:05.447320 31165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:05.447800 31160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:31:05.448916 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db7917f4c934e85837edcb7e357b616. Latest consensus state: current_term: 1 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:05.449033 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:05.449375 31166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:05.449985 31166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:05.449419 31174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:31:05.452500 31174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:31:05.457371 31174 catalog_manager.cc:1347] Generated new cluster ID: 3bce9d0b91154dd6812fa037ad65360c
I20260927 16:31:05.457432 31174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:31:05.478487 31174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:31:05.479485 31174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:31:05.483542 31174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Generated new TSK 0
I20260927 16:31:05.484226 31174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:31:05.518114 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:05.627864 31186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:05.628118 31186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:05.628161 31186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:05.632128 31186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:05.632179 31186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:05.632272 31186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:31:05.638015 31186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:05.639016 31186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:05.640331 31186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:05.649719 31191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:05.649724 31192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:05.649752 31194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:05.650148 31186 server_base.cc:1034] running on GCE node
I20260927 16:31:05.650544 31186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:05.651209 31186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:05.652386 31186 hybrid_clock.cc:648] HybridClock initialized: now 1790526665652367 us; error 48 us; skew 500 ppm
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:05.654989 31186 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:31:05.656183 31186 webserver.cc:460] Webserver started at http://127.15.93.129:35233/ using document root <none> and password file <none>
I20260927 16:31:05.656726 31186 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:05.656783 31186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:05.656944 31186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:31:05.658885 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "bb4233e8bc7139a39f4b04414d4c5986"
server_key_iv: "a2f863851e423bd1fa87c9410d995630"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.659444 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "bb4233e8bc7139a39f4b04414d4c5986"
server_key_iv: "a2f863851e423bd1fa87c9410d995630"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.662921 31186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:31:05.665189 31201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.666342 31186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:05.666463 31186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "bb4233e8bc7139a39f4b04414d4c5986"
server_key_iv: "a2f863851e423bd1fa87c9410d995630"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.666560 31186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:05.691401 31186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:05.694485 31186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:05.694674 31186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:05.695238 31186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:05.696040 31186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:05.696084 31186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.696165 31186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:05.696200 31186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.704578 31186 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:38189
I20260927 16:31:05.704975 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:38189 every 8 connection(s)
I20260927 16:31:05.705597 31186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:05.714054 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31186
I20260927 16:31:05.714181 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:31:05.714493 15734 external_mini_cluster.cc:1428] Setting key 916819c2965b1389b5612e6b676673ac
I20260927 16:31:05.718199 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:05.707243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:36573 (local address 127.15.93.190:38483)
0927 16:31:05.707406 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:05.707413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:05.708206 (+ 793us) server_negotiation.cc:407] Connection header received
0927 16:31:05.709077 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:05.709080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:05.709135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:05.709214 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:05.710324 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.710855 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.712046 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.712279 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.712385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:05.714717 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:05.714721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:05.714722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:05.714746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:05.716214 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.716724 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.716728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.716730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.716811 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.717151 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.717154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.717156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.717329 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:05.717427 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:05.717911 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 16:31:05.718034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:31:05.718982 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:05.707653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:36573)
0927 16:31:05.708054 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:05.708107 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:05.708883 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:05.709372 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:05.709379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:05.709722 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:05.710182 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:05.710194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.710977 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.710981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:05.711920 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:05.711929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.712420 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.712424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:05.712583 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:05.713086 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:05.713112 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:05.714547 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:05.716344 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:05.716350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:05.716364 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:05.716618 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:05.716904 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:05.716907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:05.716909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:05.717041 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:05.717437 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:05.717445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:05.717683 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 16:31:05.717857 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 16:31:05.718179 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260927 16:31:05.720192 31315 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:05.720422 31315 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:05.720885 31315 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:05.722506 31124 ts_manager.cc:194] Registered new tserver with Master: 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:31:05.724114 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:36573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:31:05.742286 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:05.851518 31320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:05.851809 31320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:05.851851 31320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:05.855810 31320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:05.855861 31320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:05.855919 31320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:31:05.861069 31320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:05.862059 31320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:05.863001 31320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:05.872391 31328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:05.872395 31326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:05.872628 31325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:05.872673 31320 server_base.cc:1034] running on GCE node
I20260927 16:31:05.873144 31320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:05.873765 31320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:05.874941 31320 hybrid_clock.cc:648] HybridClock initialized: now 1790526665874906 us; error 51 us; skew 500 ppm
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:05.877683 31320 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:31:05.878445 31320 webserver.cc:460] Webserver started at http://127.15.93.130:46595/ using document root <none> and password file <none>
I20260927 16:31:05.878950 31320 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:05.879006 31320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:05.879164 31320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:31:05.881155 31320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "e97f318cf97591e3ded8705c89b9e0e7"
server_key_iv: "e71ceeaa90c7c2c48d7983b18e67ad9f"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.881654 31320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "e97f318cf97591e3ded8705c89b9e0e7"
server_key_iv: "e71ceeaa90c7c2c48d7983b18e67ad9f"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.885185 31320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:31:05.887559 31335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.888787 31320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:05.888880 31320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "e97f318cf97591e3ded8705c89b9e0e7"
server_key_iv: "e71ceeaa90c7c2c48d7983b18e67ad9f"
server_key_version: "encryptionkey@0"
I20260927 16:31:05.888967 31320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:05.906292 31320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:05.909468 31320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:05.909624 31320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:05.910133 31320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:05.910884 31320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:05.910928 31320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.910966 31320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:05.910987 31320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:05.919304 31320 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34937
I20260927 16:31:05.919334 31448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34937 every 8 connection(s)
I20260927 16:31:05.920176 31320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:31:05.928369 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31320
I20260927 16:31:05.928474 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:31:05.928786 15734 external_mini_cluster.cc:1428] Setting key c3551ba6d35fbbc9f4f25a76a393cacd
Sep 27 16:31:05 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526665, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:05.933475 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:05.921907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:42883 (local address 127.15.93.190:38483)
0927 16:31:05.922064 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:05.922069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:05.922920 (+ 851us) server_negotiation.cc:407] Connection header received
0927 16:31:05.924155 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:05.924158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:05.924207 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:05.924282 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:05.925387 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.925907 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.926957 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.927162 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.927233 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:05.929719 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:05.929721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:05.929723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:05.929746 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:05.931426 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.931897 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.931903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.931908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.932007 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:05.932406 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:05.932408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:05.932409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:05.932575 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:05.932656 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:05.933265 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 16:31:05.933359 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:31:05.934279 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:05.922325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:42883)
0927 16:31:05.922750 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:05.922814 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:05.923961 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:05.924438 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:05.924444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:05.924805 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:05.925269 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:05.925282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.926004 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.926006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:05.926861 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:05.926869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:05.927287 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:05.927291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:05.927494 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:05.928016 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:05.928032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:05.929565 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:05.931530 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:05.931535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:05.931547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:05.931803 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:05.932114 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:05.932120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:05.932124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:05.932310 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:05.932679 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:05.932687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:05.933016 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 16:31:05.933235 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 16:31:05.933491 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260927 16:31:05.935406 31449 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:05.935647 31449 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:05.936118 31449 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:05.937148 31124 ts_manager.cc:194] Registered new tserver with Master: 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
I20260927 16:31:05.937965 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:42883
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:31:05.956316 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:06.062484 31454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:06.062781 31454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:06.062855 31454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:06.066851 31454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:06.066943 31454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:06.067034 31454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:31:06.072422 31454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:06.073488 31454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:06.074514 31454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:06.083865 31460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:06.083875 31459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:06.084033 31454 server_base.cc:1034] running on GCE node
W20260927 16:31:06.083957 31462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:06.084626 31454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:06.085266 31454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:06.086443 31454 hybrid_clock.cc:648] HybridClock initialized: now 1790526666086428 us; error 30 us; skew 500 ppm
Sep 27 16:31:06 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:06.089250 31454 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:31:06.090157 31454 webserver.cc:460] Webserver started at http://127.15.93.131:46797/ using document root <none> and password file <none>
I20260927 16:31:06.090716 31454 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:06.090801 31454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:06.091023 31454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:31:06.093096 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "2426f34ddd8f4f28b40068d47e1cc777"
format_stamp: "Formatted at 2026-09-27 16:31:06 on dist-test-slave-grzf"
server_key: "4c4303203c1b5a3a0eacde269efd8d13"
server_key_iv: "ae5d824e78c276aa0f094010df489487"
server_key_version: "encryptionkey@0"
I20260927 16:31:06.093655 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "2426f34ddd8f4f28b40068d47e1cc777"
format_stamp: "Formatted at 2026-09-27 16:31:06 on dist-test-slave-grzf"
server_key: "4c4303203c1b5a3a0eacde269efd8d13"
server_key_iv: "ae5d824e78c276aa0f094010df489487"
server_key_version: "encryptionkey@0"
I20260927 16:31:06.097286 31454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:31:06.099581 31469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:06.100731 31454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:06.100855 31454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "2426f34ddd8f4f28b40068d47e1cc777"
format_stamp: "Formatted at 2026-09-27 16:31:06 on dist-test-slave-grzf"
server_key: "4c4303203c1b5a3a0eacde269efd8d13"
server_key_iv: "ae5d824e78c276aa0f094010df489487"
server_key_version: "encryptionkey@0"
I20260927 16:31:06.100960 31454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:06.109768 31454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:06.112403 31454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:06.112589 31454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:06.113279 31454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:06.114087 31454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:06.114168 31454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:06.114235 31454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:06.114286 31454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:06.123009 31454 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40177
I20260927 16:31:06.123026 31582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40177 every 8 connection(s)
I20260927 16:31:06.123926 31454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:31:06.131855 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31454
I20260927 16:31:06.131994 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:31:06.132292 15734 external_mini_cluster.cc:1428] Setting key 6669290a163170102486f40cb4d7a739
Sep 27 16:31:06 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526666, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:06.137152 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.125620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:40227 (local address 127.15.93.190:38483)
0927 16:31:06.125821 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.125825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.126606 (+ 781us) server_negotiation.cc:407] Connection header received
0927 16:31:06.127848 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.127851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.127900 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.127994 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:06.129090 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.129642 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.130710 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.130918 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.131010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.133712 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:06.133715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:06.133717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:06.133741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:06.135184 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:06.135716 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:06.135720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:06.135722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:06.135798 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:06.136129 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:06.136133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:06.136135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:06.136306 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:06.136408 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.136873 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.136991 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 16:31:06.137920 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.126033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:40227)
0927 16:31:06.126443 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.126484 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.127655 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.128145 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.128151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.128487 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:06.128954 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.128966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.129744 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.129746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.130605 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.130614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.131049 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.131051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.131179 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.131807 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:06.131822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:06.133548 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:06.135312 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:06.135318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:06.135332 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:06.135619 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:06.135891 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:06.135894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:06.135896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:06.136029 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:06.136406 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:06.136413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:06.136647 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.136818 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.137132 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 16:31:06.138919 31583 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:06.139174 31583 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:06.139714 31583 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:06.140656 31124 ts_manager.cc:194] Registered new tserver with Master: 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177)
I20260927 16:31:06.141250 31124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:40227
I20260927 16:31:06.146008 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:31:06.156337 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.148139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.15.93.190:38483)
0927 16:31:06.148242 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.148246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.148638 (+ 392us) server_negotiation.cc:407] Connection header received
0927 16:31:06.148746 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.148748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.148788 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.148884 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:06.149831 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.150539 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.151915 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.152205 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.152265 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.153694 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:06.153697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:06.153699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:06.153721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:06.154929 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:06.155289 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:06.155291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:06.155292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:06.155359 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:06.155629 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:06.155632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:06.155633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:06.155825 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:06.155927 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.156140 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.156224 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:31:06.158885 31124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:31:06.161851 31124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:31:06.175684 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.169691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.15.93.131:40177)
0927 16:31:06.170069 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.170082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.170283 (+ 201us) server_negotiation.cc:407] Connection header received
0927 16:31:06.170377 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.170383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.170842 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.171125 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.171716 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.172872 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.174256 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.174802 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.174925 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175071 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.175469 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.175565 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":86,"threads_started":1}
I20260927 16:31:06.175737 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.170315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.15.93.130:34937)
0927 16:31:06.170822 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.170837 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.170859 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:31:06.170933 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.170938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.171159 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.171340 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.171688 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.172581 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.174214 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.175025 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.175176 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175313 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.175463 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.175565 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":117,"threads_started":1}
I20260927 16:31:06.176082 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.169893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.0.0.1:38066)
0927 16:31:06.170478 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.170491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.170573 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.171329 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.171335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.171360 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.171563 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.171572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.172701 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.172707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.174072 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.174084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.175192 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.175196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.175274 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175294 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.175391 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.175454 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":8,"spinlock_wait_cycles":2816,"thread_start_us":92,"threads_started":1}
I20260927 16:31:06.176203 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.170634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.15.93.129:38189)
0927 16:31:06.171561 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.171575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.171599 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:31:06.171674 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.171682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.171868 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.172013 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.172205 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.173185 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.174521 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.175304 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.175771 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175911 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.175975 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.176055 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":82,"threads_started":1}
I20260927 16:31:06.176321 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.170270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.0.0.1:37338)
0927 16:31:06.170958 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.170972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.171068 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.172005 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.172008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.172026 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.172113 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.172118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.173342 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.173346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.174366 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.174377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.175457 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.175461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.175554 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175576 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.175627 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.175685 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":82,"threads_started":1}
I20260927 16:31:06.177100 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.169511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40177 (local address 127.0.0.1:55986)
0927 16:31:06.169855 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.169906 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.170035 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.171356 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.171359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.171403 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.171574 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.171583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.172998 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.173001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.174113 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.174124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.174924 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.174928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.175031 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.175065 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.175188 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.175304 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":22,"thread_start_us":100,"threads_started":1}
I20260927 16:31:06.179041 31249 tablet_service.cc:1467] Processing CreateTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a (DEFAULT_TABLE table=test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:06.179075 31383 tablet_service.cc:1467] Processing CreateTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a (DEFAULT_TABLE table=test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:06.180034 31249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a3649bce754cd3b0223cd3c52a8b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:06.180071 31383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a3649bce754cd3b0223cd3c52a8b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:06.179805 31517 tablet_service.cc:1467] Processing CreateTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a (DEFAULT_TABLE table=test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:06.180685 31517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a3649bce754cd3b0223cd3c52a8b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:06.185860 31600 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap starting.
I20260927 16:31:06.186818 31601 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Bootstrap starting.
I20260927 16:31:06.188062 31602 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Bootstrap starting.
I20260927 16:31:06.188235 31600 tablet_bootstrap.cc:654] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:06.189356 31601 tablet_bootstrap.cc:654] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:06.189579 31600 log.cc:824] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:06.190032 31602 tablet_bootstrap.cc:654] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:06.190187 31601 log.cc:824] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:06.191198 31602 log.cc:824] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:06.191646 31600 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: No bootstrap required, opened a new log
I20260927 16:31:06.191848 31600 ts_tablet_manager.cc:1397] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:31:06.192145 31601 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: No bootstrap required, opened a new log
I20260927 16:31:06.192312 31601 ts_tablet_manager.cc:1397] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:31:06.193059 31602 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: No bootstrap required, opened a new log
I20260927 16:31:06.193240 31602 ts_tablet_manager.cc:1397] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:31:06.195832 31600 raft_consensus.cc:348] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.195899 31601 raft_consensus.cc:348] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.196108 31600 raft_consensus.cc:374] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:06.196110 31601 raft_consensus.cc:374] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:06.196173 31601 raft_consensus.cc:729] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376aeed9ae8c4f0494c2f8fbb9d40527, State: Initialized, Role: FOLLOWER
I20260927 16:31:06.196199 31600 raft_consensus.cc:729] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63af3b5cd0e54c3c8573eb06bbfac1e0, State: Initialized, Role: FOLLOWER
I20260927 16:31:06.196342 31602 raft_consensus.cc:348] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.196594 31602 raft_consensus.cc:374] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:06.196679 31602 raft_consensus.cc:729] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2426f34ddd8f4f28b40068d47e1cc777, State: Initialized, Role: FOLLOWER
I20260927 16:31:06.196663 31601 consensus_queue.cc:260] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.196663 31600 consensus_queue.cc:260] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.197304 31449 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
I20260927 16:31:06.197551 31315 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
I20260927 16:31:06.197649 31601 ts_tablet_manager.cc:1428] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:31:06.197104 31602 consensus_queue.cc:260] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.198549 31600 ts_tablet_manager.cc:1428] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 16:31:06.198987 31602 ts_tablet_manager.cc:1428] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:31:06.199079 31583 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
W20260927 16:31:06.206637 31316 tablet.cc:2369] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:31:06.309157 31606 raft_consensus.cc:482] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:06.309355 31606 raft_consensus.cc:504] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.310423 31606 leader_election.cc:288] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
I20260927 16:31:06.315194 31607 raft_consensus.cc:482] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:06.315441 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.311184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:55747 (local address 127.15.93.129:38189)
0927 16:31:06.311322 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.311327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.311483 (+ 156us) server_negotiation.cc:407] Connection header received
0927 16:31:06.311603 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.311606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.311674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.311752 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.311979 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.312657 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314093 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.314748 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314876 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.314924 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.315205 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.315298 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:31:06.315429 31609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.311086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.15.93.130:55747)
0927 16:31:06.311397 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.311410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.311503 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.311769 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.311771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.311787 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.311880 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.311884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.312754 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.312757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.313951 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.313957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.314882 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.315023 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.315055 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.315156 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.315233 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":42,"thread_start_us":120,"threads_started":1}
I20260927 16:31:06.315583 31607 raft_consensus.cc:504] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.315709 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.310867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:60587 (local address 127.15.93.131:40177)
0927 16:31:06.310996 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.311001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.311017 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:31:06.311145 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.311148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.311194 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.311277 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.311676 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.312465 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314093 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.314724 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314871 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.314924 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.315484 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.315564 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:31:06.316031 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.310755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40177 (local address 127.15.93.130:60587)
0927 16:31:06.310879 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.310892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.311016 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.311374 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.311393 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.311419 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.311567 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.311573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.312587 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.312590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.313943 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.313955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.314882 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.314886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.315211 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.315230 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.315298 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.315355 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":195}
I20260927 16:31:06.316123 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" is_pre_election: true
I20260927 16:31:06.316380 31269 raft_consensus.cc:2454] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376aeed9ae8c4f0494c2f8fbb9d40527 in term 0.
I20260927 16:31:06.316661 31537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2426f34ddd8f4f28b40068d47e1cc777" is_pre_election: true
I20260927 16:31:06.316921 31337 leader_election.cc:302] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376aeed9ae8c4f0494c2f8fbb9d40527, 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters:
I20260927 16:31:06.316936 31537 raft_consensus.cc:2454] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376aeed9ae8c4f0494c2f8fbb9d40527 in term 0.
I20260927 16:31:06.317338 31606 raft_consensus.cc:2781] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:31:06.317441 31606 raft_consensus.cc:482] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:31:06.317512 31606 raft_consensus.cc:3037] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:06.317788 31607 leader_election.cc:288] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
I20260927 16:31:06.318964 31606 raft_consensus.cc:504] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.319331 31606 leader_election.cc:288] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [CANDIDATE]: Term 1 election: Requested vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
I20260927 16:31:06.319729 31537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2426f34ddd8f4f28b40068d47e1cc777"
I20260927 16:31:06.319736 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
I20260927 16:31:06.319844 31269 raft_consensus.cc:3037] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:06.319846 31537 raft_consensus.cc:3037] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:06.321082 31269 raft_consensus.cc:2454] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376aeed9ae8c4f0494c2f8fbb9d40527 in term 1.
I20260927 16:31:06.321177 31537 raft_consensus.cc:2454] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376aeed9ae8c4f0494c2f8fbb9d40527 in term 1.
I20260927 16:31:06.321573 31337 leader_election.cc:302] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527; no voters:
I20260927 16:31:06.321717 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.317266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:46395 (local address 127.15.93.131:40177)
0927 16:31:06.317376 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.317380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.317543 (+ 163us) server_negotiation.cc:407] Connection header received
0927 16:31:06.317656 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.317661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.317704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.318192 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.318294 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.318961 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.320412 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.320978 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.321081 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.321142 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.321483 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.321572 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:31:06.321993 31606 raft_consensus.cc:2781] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:31:06.322103 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.317069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40177 (local address 127.15.93.129:46395)
0927 16:31:06.317440 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.317456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.317555 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.317808 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.317812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.317835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.317965 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.317972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.319060 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.319065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.320275 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.320285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.321243 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.321246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.321337 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.321366 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.321872 (+ 506us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.321946 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":96,"threads_started":1}
I20260927 16:31:06.322147 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.318080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:48387 (local address 127.15.93.130:34937)
0927 16:31:06.318183 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.318188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.318324 (+ 136us) server_negotiation.cc:407] Connection header received
0927 16:31:06.318419 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.318422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.318480 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.318565 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:06.318820 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.319519 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.320931 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.321478 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.321573 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.321623 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.321961 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.322031 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 16:31:06.322394 31606 raft_consensus.cc:686] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 LEADER]: Becoming Leader. State: Replica: 376aeed9ae8c4f0494c2f8fbb9d40527, State: Running, Role: LEADER
I20260927 16:31:06.322503 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.317965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.15.93.129:48387)
0927 16:31:06.318234 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.318248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:06.318345 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:06.318587 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:06.318590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:06.318615 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:06.318713 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.318719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.319643 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.319646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.320788 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:06.320798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.321713 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.321717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:06.321814 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.321837 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:31:06.322261 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 16:31:06.322339 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260927 16:31:06.322666 31537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2426f34ddd8f4f28b40068d47e1cc777" is_pre_election: true
I20260927 16:31:06.322769 31606 consensus_queue.cc:237] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:06.322872 31537 raft_consensus.cc:2379] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63af3b5cd0e54c3c8573eb06bbfac1e0 in current term 1: Already voted for candidate 376aeed9ae8c4f0494c2f8fbb9d40527 in this term.
I20260927 16:31:06.323141 31403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a3649bce754cd3b0223cd3c52a8b6a" candidate_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" is_pre_election: true
I20260927 16:31:06.323791 31202 leader_election.cc:302] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:06.324061 31607 raft_consensus.cc:2726] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 16:31:06.326028 31122 catalog_manager.cc:5495] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130). New cstate: current_term: 1 leader_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } health_report { overall_health: HEALTHY } } }
I20260927 16:31:06.370312 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.366891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.15.93.130:34937)
0927 16:31:06.367012 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:06.367015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:06.367028 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:31:06.367138 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:06.367140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:06.367196 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:06.367285 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:31:06.367646 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.368161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.368972 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:06.369187 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:06.369269 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:06.369512 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:31:06.369927 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:31:06.370034 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:06.370083 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 16:31:06.370129 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:31:06.374833 31584 tablet.cc:2369] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:31:06.376240 31537 raft_consensus.cc:1261] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Refusing update from remote peer 376aeed9ae8c4f0494c2f8fbb9d40527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:31:06.376283 31269 raft_consensus.cc:1261] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Refusing update from remote peer 376aeed9ae8c4f0494c2f8fbb9d40527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:31:06.376847 31612 consensus_queue.cc:1035] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:31:06.377049 31606 consensus_queue.cc:1035] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:31:06.382753 31615 mvcc.cc:205] Tried to move back new op lower bound from 7333997225469763584 to 7333997225261207552. Current Snapshot: MvccSnapshot[applied={T|T < 7333997225469763584}]
I20260927 16:31:06.385350 31616 mvcc.cc:205] Tried to move back new op lower bound from 7333997225469763584 to 7333997225261207552. Current Snapshot: MvccSnapshot[applied={T|T < 7333997225469763584}]
I20260927 16:31:06.386808 31617 mvcc.cc:205] Tried to move back new op lower bound from 7333997225469763584 to 7333997225261207552. Current Snapshot: MvccSnapshot[applied={T|T < 7333997225469763584 or (T in {7333997225469763584})}]
I20260927 16:31:06.391300 31124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:31:06.391530 31124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:31:06.394052 31124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Sending DeleteTablet for 3 replicas of tablet 06a3649bce754cd3b0223cd3c52a8b6a
I20260927 16:31:06.394788 31383 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:06 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38066
I20260927 16:31:06.394868 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31186
I20260927 16:31:06.394815 31517 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:06 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:55986
I20260927 16:31:06.396873 31626 tablet_replica.cc:330] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: stopping tablet replica
I20260927 16:31:06.397224 31626 raft_consensus.cc:2229] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:31:06.397558 31626 raft_consensus.cc:2258] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:06.399084 31626 ts_tablet_manager.cc:1905] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:31:06.402402 31625 tablet_replica.cc:330] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: stopping tablet replica
I20260927 16:31:06.402436 31626 ts_tablet_manager.cc:1918] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:31:06.402621 31626 log.cc:1196] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/06a3649bce754cd3b0223cd3c52a8b6a
I20260927 16:31:06.402745 31625 raft_consensus.cc:2229] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 LEADER]: Raft consensus shutting down.
W20260927 16:31:06.402911 31109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:38189 (error 108)
I20260927 16:31:06.403083 31626 ts_tablet_manager.cc:1939] T 06a3649bce754cd3b0223cd3c52a8b6a P 2426f34ddd8f4f28b40068d47e1cc777: Deleting consensus metadata
I20260927 16:31:06.403126 31625 raft_consensus.cc:2258] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:06.403851 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31320
I20260927 16:31:06.405395 31109 catalog_manager.cc:4841] TS 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): tablet 06a3649bce754cd3b0223cd3c52a8b6a (table test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]) successfully deleted
I20260927 16:31:06.405522 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.405236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.0.0.1:37342)
0927 16:31:06.405366 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.405441 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:38189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 16:31:06.405756 31109 catalog_manager.cc:4584] TS 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06a3649bce754cd3b0223cd3c52a8b6a: Network error: Client connection negotiation failed: client connection to 127.15.93.129:38189: connect: Connection refused (error 111)
I20260927 16:31:06.410637 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31454
I20260927 16:31:06.410831 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:06.410504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.0.0.1:38096)
0927 16:31:06.410659 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:06.410741 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.130:34937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 16:31:06.416370 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31092
I20260927 16:31:26.422775 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--webserver_interface=127.15.93.190
--webserver_port=38083
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:26.540632 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:26.541064 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:26.541185 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:26.549206 31628 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:31:26.549293 31628 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:31:26.549321 31628 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:31:26.549341 31628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:31:26.549371 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:26.549408 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:26.549453 31628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:31:26.549487 31628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:31:26.558743 31628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=38083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:26.560217 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:26.561713 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:26.576963 31637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.576843 31634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.576843 31635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:26.577072 31628 server_base.cc:1034] running on GCE node
I20260927 16:31:26.578030 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:26.579335 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:26.580544 31628 hybrid_clock.cc:648] HybridClock initialized: now 1790526686580509 us; error 56 us; skew 500 ppm
Sep 27 16:31:26 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:26.583290 31628 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:31:26.584273 31628 webserver.cc:460] Webserver started at http://127.15.93.190:38083/ using document root <none> and password file <none>
I20260927 16:31:26.584848 31628 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:26.584952 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:26.588867 31628 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:31:26.590901 31644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:26.592195 31628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:26.592330 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "1db7917f4c934e85837edcb7e357b616"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "484bbffe1f907e5a9efb162f4d7342d2"
server_key_iv: "5308c679e9af3ebda9abd451c6df6bf7"
server_key_version: "encryptionkey@0"
I20260927 16:31:26.592921 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:26.613484 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:26.616899 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:26.617111 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:26.624240 31628 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38483
I20260927 16:31:26.624529 31696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38483 every 8 connection(s)
I20260927 16:31:26.625432 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:31:26.629596 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31628
I20260927 16:31:26.629894 31697 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:31:26.630096 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:38189
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=35233
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:31:26.632009 31697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap starting.
I20260927 16:31:26.649259 31697 log.cc:824] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:26.657735 31697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:31:26.658118 31697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap complete.
I20260927 16:31:26.662240 31697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:26.662665 31697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Initialized, Role: FOLLOWER
I20260927 16:31:26.663196 31697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:26.663342 31697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:31:26.663466 31697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:31:26.663583 31697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 1 FOLLOWER]: Advancing to term 2
I20260927 16:31:26.665452 31697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:26.665792 31697 leader_election.cc:302] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db7917f4c934e85837edcb7e357b616; no voters:
I20260927 16:31:26.666131 31697 leader_election.cc:288] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 16:31:26.666237 31701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 16:31:26.666498 31701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 LEADER]: Becoming Leader. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Running, Role: LEADER
I20260927 16:31:26.666858 31701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:26.667405 31697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:31:26.667877 31703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db7917f4c934e85837edcb7e357b616. Latest consensus state: current_term: 2 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:26.667987 31703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:26.668263 31702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:26.668367 31702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:26.668622 31705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:31:26.673409 31705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:31:26.674086 31705 catalog_manager.cc:1259] Loaded cluster ID: 3bce9d0b91154dd6812fa037ad65360c
I20260927 16:31:26.674166 31705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:31:26.676692 31705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:31:26.677383 31705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Loaded TSK: 0
I20260927 16:31:26.692696 31705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Generated new TSK 1
I20260927 16:31:26.692981 31705 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:31:26.745119 31699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:26.745419 31699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:26.745496 31699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:26.749866 31699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:26.749958 31699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:26.750054 31699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:31:26.755307 31699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:38189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=35233
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:26.756492 31699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:26.757522 31699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:26.768492 31727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.768541 31724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.768518 31725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:26.769131 31699 server_base.cc:1034] running on GCE node
I20260927 16:31:26.769513 31699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:26.770202 31699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:26.771420 31699 hybrid_clock.cc:648] HybridClock initialized: now 1790526686771388 us; error 49 us; skew 500 ppm
Sep 27 16:31:26 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:26.774783 31699 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:31:26.775846 31699 webserver.cc:460] Webserver started at http://127.15.93.129:35233/ using document root <none> and password file <none>
I20260927 16:31:26.776495 31699 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:26.776573 31699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:26.780537 31699 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:31:26.782701 31734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:26.784017 31699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:31:26.784145 31699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "bb4233e8bc7139a39f4b04414d4c5986"
server_key_iv: "a2f863851e423bd1fa87c9410d995630"
server_key_version: "encryptionkey@0"
I20260927 16:31:26.784691 31699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:26.796667 31699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:26.799973 31699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:26.800179 31699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:26.800815 31699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:26.801959 31741 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:31:26.806138 31699 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:31:26.806202 31699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 16:31:26.806248 31699 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:31:26.808038 31699 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:31:26.808094 31699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 16:31:26.808342 31741 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap starting.
I20260927 16:31:26.819226 31699 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:38189
I20260927 16:31:26.819557 31848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:38189 every 8 connection(s)
I20260927 16:31:26.820564 31699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:31:26.827039 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31699
I20260927 16:31:26.827735 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:34937
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=46595
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:31:26.830127 31741 log.cc:824] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Log is configured to *not* fsync() on all Append() calls
Sep 27 16:31:26 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:26.840023 31741 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:31:26.840560 31741 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap complete.
I20260927 16:31:26.842134 31741 ts_tablet_manager.cc:1397] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.012s
I20260927 16:31:26.844821 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:26.823088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:35849)
0927 16:31:26.823724 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:26.823793 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:26.825017 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:26.829000 (+ 3983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:26.829010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:26.829531 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:26.830248 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:26.830269 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.832702 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.832706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:26.834168 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:26.834180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.835328 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.835332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:26.835586 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:26.836327 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:26.836348 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:26.838427 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:26.841120 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:26.841126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:26.841140 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:26.841479 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:26.841918 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:26.841921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:26.841924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:26.842101 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:26.842882 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:26.842889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:26.843207 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 16:31:26.843460 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 16:31:26.843842 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":199,"threads_started":1}
I20260927 16:31:26.844822 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:26.823092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:35849 (local address 127.15.93.190:38483)
0927 16:31:26.827586 (+ 4494us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:26.827608 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:26.827661 (+ 53us) server_negotiation.cc:407] Connection header received
0927 16:31:26.827982 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:26.828000 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:26.828457 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:26.830537 (+ 2080us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:26.830840 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.832505 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.834345 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.835153 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.835997 (+ 844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:26.838704 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:26.838711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:26.838727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:26.838761 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:26.840970 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:26.841644 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:26.841650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:26.841656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:26.841800 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:26.842227 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:26.842230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:26.842232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:26.842729 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:26.843474 (+ 745us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:26.843631 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 16:31:26.843910 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4301,"thread_start_us":109,"threads_started":1}
I20260927 16:31:26.845570 31741 raft_consensus.cc:348] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:26.846104 31741 raft_consensus.cc:729] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63af3b5cd0e54c3c8573eb06bbfac1e0, State: Initialized, Role: FOLLOWER
I20260927 16:31:26.846661 31741 consensus_queue.cc:260] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:26.847627 31849 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:26.847709 31741 ts_tablet_manager.cc:1428] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:31:26.847880 31849 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:26.848393 31849 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:26.850894 31661 ts_manager.cc:194] Registered new tserver with Master: 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
I20260927 16:31:26.854596 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:35849
I20260927 16:31:26.857468 31849 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
Sep 27 16:31:26 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.129@KRBTEST.COM
I20260927 16:31:26.864625 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:26.854048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.15.93.129:38189)
0927 16:31:26.854760 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:26.854772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:26.854795 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:31:26.854864 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:26.854870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:26.855069 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:26.855222 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:26.856091 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.856959 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.858092 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.858460 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.858589 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:26.861064 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:26.861074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:26.861094 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:26.861116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:26.862672 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:26.863179 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:26.863185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:26.863187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:26.863274 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:26.863596 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:26.863599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:26.863601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:26.863836 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:26.863975 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:26.864310 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 16:31:26.864472 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":618,"thread_start_us":84,"threads_started":1}
I20260927 16:31:26.864617 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:26.853943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.0.0.1:38614)
0927 16:31:26.854382 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:26.854451 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:26.854598 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:26.855251 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:26.855257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:26.855812 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:26.855951 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:26.855959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.857101 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.857109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:26.857958 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:26.857970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:26.858592 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:26.858596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:26.858729 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:26.859449 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:26.859468 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:26.860906 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:26.862799 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:26.862810 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:26.862815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:26.863046 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:26.863403 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:26.863406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:26.863408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:26.863492 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:26.863950 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:26.863956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:26.864080 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 16:31:26.864217 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 16:31:26.864416 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260927 16:31:26.865828 31783 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:06 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38614
I20260927 16:31:26.866830 31858 tablet_replica.cc:330] stopping tablet replica
I20260927 16:31:26.867203 31858 raft_consensus.cc:2229] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:31:26.867504 31858 raft_consensus.cc:2258] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:26.868634 31858 ts_tablet_manager.cc:1905] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:31:26.872273 31858 ts_tablet_manager.cc:1918] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:31:26.872354 31858 log.cc:1196] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal/wals/06a3649bce754cd3b0223cd3c52a8b6a
I20260927 16:31:26.872628 31858 ts_tablet_manager.cc:1939] T 06a3649bce754cd3b0223cd3c52a8b6a P 63af3b5cd0e54c3c8573eb06bbfac1e0: Deleting consensus metadata
I20260927 16:31:26.874306 31646 catalog_manager.cc:4841] TS 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189): tablet 06a3649bce754cd3b0223cd3c52a8b6a (table test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]) successfully deleted
W20260927 16:31:26.950829 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:26.951102 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:26.951144 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:26.955171 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:26.955231 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:26.955291 31853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:31:26.960697 31853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:34937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=46595
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:26.961714 31853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:26.962677 31853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:26.972828 31866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.972815 31864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:26.972867 31863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:26.973004 31853 server_base.cc:1034] running on GCE node
I20260927 16:31:26.973584 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:26.974218 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:26.975455 31853 hybrid_clock.cc:648] HybridClock initialized: now 1790526686975419 us; error 59 us; skew 500 ppm
Sep 27 16:31:26 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:26.978129 31853 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:31:26.979032 31853 webserver.cc:460] Webserver started at http://127.15.93.130:46595/ using document root <none> and password file <none>
I20260927 16:31:26.979679 31853 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:26.979790 31853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:26.983817 31853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:31:26.986129 31873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:26.987421 31853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:26.987568 31853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "e97f318cf97591e3ded8705c89b9e0e7"
server_key_iv: "e71ceeaa90c7c2c48d7983b18e67ad9f"
server_key_version: "encryptionkey@0"
I20260927 16:31:26.988173 31853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:27.003680 31853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:27.006774 31853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:27.006989 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:27.007647 31853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:27.008718 31880 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:31:27.012902 31853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:31:27.013003 31853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 16:31:27.013085 31853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:31:27.014817 31853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:31:27.014894 31853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 16:31:27.015210 31880 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Bootstrap starting.
I20260927 16:31:27.025058 31853 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34937
I20260927 16:31:27.025084 31987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34937 every 8 connection(s)
I20260927 16:31:27.026015 31853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:31:27.035277 31880 log.cc:824] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:27.036301 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31853
Sep 27 16:31:27 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526686, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:27.036790 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:40177
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=46797
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:31:27.041709 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.027937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:42299 (local address 127.15.93.190:38483)
0927 16:31:27.028090 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.028095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.028867 (+ 772us) server_negotiation.cc:407] Connection header received
0927 16:31:27.030178 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.030181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.030231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.030307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:27.031990 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.032509 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.034231 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.034465 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.034543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.037490 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:27.037493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:27.037495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:27.037520 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:27.039458 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.040060 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.040063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.040064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.040180 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.040557 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.040563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.040567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.040770 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:27.040881 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.041379 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.041532 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:31:27.042582 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.028078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:42299)
0927 16:31:27.028682 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.028744 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.029901 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.030497 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.030506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.030993 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:27.031830 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.031850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.032675 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.032682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.034110 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.034121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.034576 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.034580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.034776 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.035418 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:27.035439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:27.037285 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:27.039587 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:27.039600 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:27.039615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:27.039944 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:27.040287 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:27.040291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:27.040293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:27.040448 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:27.040881 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:27.040889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:27.041139 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.041324 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.041653 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":129,"threads_started":1}
I20260927 16:31:27.043915 31988 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:27.044191 31988 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:27.044682 31988 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:27.044800 31880 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:31:27.045281 31880 tablet_bootstrap.cc:492] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Bootstrap complete.
I20260927 16:31:27.046460 31661 ts_manager.cc:194] Registered new tserver with Master: 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
I20260927 16:31:27.046819 31880 ts_tablet_manager.cc:1397] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.007s
I20260927 16:31:27.047829 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:42299
I20260927 16:31:27.050515 31880 raft_consensus.cc:348] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:27.051015 31880 raft_consensus.cc:729] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376aeed9ae8c4f0494c2f8fbb9d40527, State: Initialized, Role: FOLLOWER
I20260927 16:31:27.051611 31880 consensus_queue.cc:260] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } }
I20260927 16:31:27.052289 31988 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
I20260927 16:31:27.052610 31880 ts_tablet_manager.cc:1428] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 16:31:27.053707 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.047422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.0.0.1:48344)
0927 16:31:27.047697 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.047711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.047786 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.050149 (+ 2363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.050152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.050169 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:27.050266 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.050273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.051290 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.051293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.052487 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.052497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.053265 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.053269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.053361 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.053403 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.053481 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.053578 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260927 16:31:27.054250 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.047932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.15.93.130:34937)
0927 16:31:27.048526 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.049761 (+ 1235us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.049781 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:31:27.049833 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.049836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.049982 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.050155 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:27.050385 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.051180 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.052621 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.053138 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.053261 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.053396 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.054065 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.054159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":121,"threads_started":1}
I20260927 16:31:27.055263 31922 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a3649bce754cd3b0223cd3c52a8b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:06 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260927 16:31:27.059614 31995 tablet_replica.cc:330] stopping tablet replica
I20260927 16:31:27.059921 31995 raft_consensus.cc:2229] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:31:27.060165 31995 raft_consensus.cc:2258] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:27.061117 31995 ts_tablet_manager.cc:1905] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:31:27.064736 31995 ts_tablet_manager.cc:1918] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:31:27.064850 31995 log.cc:1196] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/06a3649bce754cd3b0223cd3c52a8b6a
I20260927 16:31:27.065127 31995 ts_tablet_manager.cc:1939] T 06a3649bce754cd3b0223cd3c52a8b6a P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting consensus metadata
I20260927 16:31:27.066682 31645 catalog_manager.cc:4841] TS 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): tablet 06a3649bce754cd3b0223cd3c52a8b6a (table test-table [id=e194d0694fe64d5e89bdb4c1bda2221d]) successfully deleted
W20260927 16:31:27.152928 31992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:27.153200 31992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:27.153285 31992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:27.157305 31992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:27.157366 31992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:27.157464 31992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:31:27.162695 31992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:40177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=46797
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:27.163806 31992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:27.164784 31992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:27.174719 32001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:27.174727 32003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:27.174727 32000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:27.174945 31992 server_base.cc:1034] running on GCE node
I20260927 16:31:27.175433 31992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:27.176061 31992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:27.177253 31992 hybrid_clock.cc:648] HybridClock initialized: now 1790526687177202 us; error 69 us; skew 500 ppm
Sep 27 16:31:27 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:27.179976 31992 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:31:27.180769 31992 webserver.cc:460] Webserver started at http://127.15.93.131:46797/ using document root <none> and password file <none>
I20260927 16:31:27.181283 31992 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:27.181350 31992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:27.185020 31992 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 16:31:27.187028 32010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:27.188359 31992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 16:31:27.188473 31992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "2426f34ddd8f4f28b40068d47e1cc777"
format_stamp: "Formatted at 2026-09-27 16:31:06 on dist-test-slave-grzf"
server_key: "4c4303203c1b5a3a0eacde269efd8d13"
server_key_iv: "ae5d824e78c276aa0f094010df489487"
server_key_version: "encryptionkey@0"
I20260927 16:31:27.188993 31992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:27.207484 31992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:27.210379 31992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:27.210583 31992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:27.211153 31992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:27.212000 31992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:27.212044 31992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:27.212117 31992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:27.212155 31992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:27.221220 31992 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40177
I20260927 16:31:27.221242 32123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40177 every 8 connection(s)
I20260927 16:31:27.222214 31992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:31:27.224581 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 31992
Sep 27 16:31:27 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526687, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:31:27.236339 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.225338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.15.93.190:38483)
0927 16:31:27.225598 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.225616 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.225855 (+ 239us) server_negotiation.cc:407] Connection header received
0927 16:31:27.225971 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.225975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.226028 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.226130 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:27.226969 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.227803 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.229110 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.229455 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.229563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.231480 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:27.231484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:27.231487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:27.231512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:27.233459 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.233988 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.233992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.233994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.234085 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.234774 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.234780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.234782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.235022 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:27.235266 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.235978 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.236127 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"spinlock_wait_cycles":1792,"thread_start_us":94,"threads_started":1}
I20260927 16:31:27.238930 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.224079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:43171 (local address 127.15.93.190:38483)
0927 16:31:27.224238 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.224242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.225487 (+ 1245us) server_negotiation.cc:407] Connection header received
0927 16:31:27.226619 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.226623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.226671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.226759 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:27.228310 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.229110 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.231198 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.231615 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.231722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.234840 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:27.234844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:27.234846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:27.234884 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:27.236735 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.237268 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.237272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.237273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.237363 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.237775 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.237778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.237780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.237975 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:27.238074 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.238729 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.238819 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":5632}
I20260927 16:31:27.239832 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.224515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:43171)
0927 16:31:27.225309 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.225367 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.226414 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.226952 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.226961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.227503 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:27.228166 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.228182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.229252 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.229256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.231017 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.231028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.231879 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.231883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.232095 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.232679 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:27.232704 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:27.234655 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:27.236863 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:27.236872 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:27.236893 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:27.237187 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:27.237511 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:27.237514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:27.237516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:27.237671 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:27.238216 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:27.238225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:27.238512 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.238675 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.238949 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":125,"threads_started":1}
I20260927 16:31:27.240959 32124 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38483
I20260927 16:31:27.241178 32124 heartbeater.cc:464] Registering TS with master...
I20260927 16:31:27.241632 32124 heartbeater.cc:511] Master 127.15.93.190:38483 requested a full tablet report, sending...
I20260927 16:31:27.242553 31661 ts_manager.cc:194] Registered new tserver with Master: 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177)
I20260927 16:31:27.243110 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:43171
I20260927 16:31:27.250114 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:31:27.254240 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.250786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.15.93.190:38483)
0927 16:31:27.250919 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.250923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.251112 (+ 189us) server_negotiation.cc:407] Connection header received
0927 16:31:27.251247 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.251249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.251348 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.251516 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:31:27.251767 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.252278 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.253125 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.253365 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.253449 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.253776 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:31:27.253913 (+ 137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 16:31:27.254139 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44980: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 16:31:27.254297 31851 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44980: token expired
W20260927 16:31:27.254655 31588 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.93.190:38483)
I20260927 16:31:27.261682 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.255008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.15.93.190:38483)
0927 16:31:27.255127 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.255130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.255167 (+ 37us) server_negotiation.cc:407] Connection header received
0927 16:31:27.255263 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.255265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.255296 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.255357 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:27.255999 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.256468 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.257256 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.257451 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.257522 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.258767 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:27.258770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:27.258771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:27.258792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:27.260396 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.260769 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.260771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.260772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.260841 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:27.261052 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:27.261054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:27.261055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:27.261218 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:27.261313 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.261480 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.261563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:31:27.264237 31661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:31:27.267494 31661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:31:27.274837 31922 tablet_service.cc:1467] Processing CreateTablet for tablet 660b75f3850d4401b60379f2c005ddb1 (DEFAULT_TABLE table=test-table [id=27fc8e2a77254f09904bc500cde8cf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:27.274891 31783 tablet_service.cc:1467] Processing CreateTablet for tablet 660b75f3850d4401b60379f2c005ddb1 (DEFAULT_TABLE table=test-table [id=27fc8e2a77254f09904bc500cde8cf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:27.275501 31922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b75f3850d4401b60379f2c005ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:27.275522 31783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b75f3850d4401b60379f2c005ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:27.278224 31741 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap starting.
I20260927 16:31:27.278332 31880 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Bootstrap starting.
I20260927 16:31:27.279074 31741 tablet_bootstrap.cc:654] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:27.279198 31880 tablet_bootstrap.cc:654] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:27.280086 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.274095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40177 (local address 127.0.0.1:40540)
0927 16:31:27.274839 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.274855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.274938 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.275786 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.275790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.275809 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:27.275918 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.275925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.277284 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.277288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.278416 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.278427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.279655 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.279659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.279769 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.279800 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.279853 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.279939 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664}
I20260927 16:31:27.280789 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.274366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.15.93.131:40177)
0927 16:31:27.275217 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.275232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.275258 (+ 26us) server_negotiation.cc:407] Connection header received
0927 16:31:27.275336 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.275342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.275629 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.275800 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:27.276049 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.277161 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.278554 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.279481 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.280270 (+ 789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.280437 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.280494 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.280614 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":229,"threads_started":1}
I20260927 16:31:27.280946 31880 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: No bootstrap required, opened a new log
I20260927 16:31:27.281023 31880 ts_tablet_manager.cc:1397] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 16:31:27.281227 31741 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: No bootstrap required, opened a new log
I20260927 16:31:27.281298 31741 ts_tablet_manager.cc:1397] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 16:31:27.281486 31880 raft_consensus.cc:348] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.281589 31880 raft_consensus.cc:374] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:27.281636 31880 raft_consensus.cc:729] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376aeed9ae8c4f0494c2f8fbb9d40527, State: Initialized, Role: FOLLOWER
I20260927 16:31:27.281728 31741 raft_consensus.cc:348] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.281759 31880 consensus_queue.cc:260] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.281831 31741 raft_consensus.cc:374] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:27.281874 31741 raft_consensus.cc:729] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63af3b5cd0e54c3c8573eb06bbfac1e0, State: Initialized, Role: FOLLOWER
I20260927 16:31:27.282006 31741 consensus_queue.cc:260] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.282258 31741 ts_tablet_manager.cc:1428] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 16:31:27.282943 31880 ts_tablet_manager.cc:1428] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 16:31:27.283923 32058 tablet_service.cc:1467] Processing CreateTablet for tablet 660b75f3850d4401b60379f2c005ddb1 (DEFAULT_TABLE table=test-table [id=27fc8e2a77254f09904bc500cde8cf25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:31:27.285034 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b75f3850d4401b60379f2c005ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:31:27.291890 32136 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Bootstrap starting.
I20260927 16:31:27.294358 32136 tablet_bootstrap.cc:654] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Neither blocks nor log segments found. Creating new log.
I20260927 16:31:27.295094 32136 log.cc:824] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:27.297015 32136 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: No bootstrap required, opened a new log
I20260927 16:31:27.297224 32136 ts_tablet_manager.cc:1397] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 16:31:27.300689 32136 raft_consensus.cc:348] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.300913 32136 raft_consensus.cc:374] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:31:27.300985 32136 raft_consensus.cc:729] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2426f34ddd8f4f28b40068d47e1cc777, State: Initialized, Role: FOLLOWER
I20260927 16:31:27.301389 32136 consensus_queue.cc:260] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.302017 32124 heartbeater.cc:503] Master 127.15.93.190:38483 was elected leader, sending a full tablet report...
I20260927 16:31:27.302351 32136 ts_tablet_manager.cc:1428] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260927 16:31:27.322053 31850 tablet.cc:2369] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:31:27.346746 32138 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:27.347033 32138 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.348388 32138 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937), 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
I20260927 16:31:27.353216 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.348709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:40305 (local address 127.15.93.130:34937)
0927 16:31:27.348823 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.348829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.349105 (+ 276us) server_negotiation.cc:407] Connection header received
0927 16:31:27.349183 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.349186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.349235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.349310 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:27.349718 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.350498 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.352288 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.352705 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.352800 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.352841 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.353020 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.353114 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:31:27.353245 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.348651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.15.93.131:40305)
0927 16:31:27.349003 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.349016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.349108 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.349326 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.349328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.349366 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:27.349576 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.349583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.350621 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.350623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.352174 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.352185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.352812 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.352815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.352890 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.352911 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.353001 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.353073 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":228,"spinlock_wait_cycles":30464,"thread_start_us":121,"threads_started":1}
I20260927 16:31:27.353839 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.348619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.15.93.131:47073)
0927 16:31:27.348805 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.348821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:27.348949 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:27.349325 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:27.349327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:27.349367 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:31:27.349578 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.349583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.350618 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.350621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.352174 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:27.352185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.353546 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.353549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:27.353630 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.353648 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:31:27.353698 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 16:31:27.353749 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":154,"spinlock_wait_cycles":5504}
I20260927 16:31:27.353919 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.348734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:47073 (local address 127.15.93.129:38189)
0927 16:31:27.348861 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.348865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.348883 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:31:27.349091 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.349096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.349178 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.349286 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:31:27.349718 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.350502 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.352891 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.353438 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.353552 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.353702 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.353733 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.353791 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:31:27.354493 31942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b75f3850d4401b60379f2c005ddb1" candidate_uuid: "2426f34ddd8f4f28b40068d47e1cc777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" is_pre_election: true
I20260927 16:31:27.354492 31803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b75f3850d4401b60379f2c005ddb1" candidate_uuid: "2426f34ddd8f4f28b40068d47e1cc777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" is_pre_election: true
I20260927 16:31:27.354796 31803 raft_consensus.cc:2454] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2426f34ddd8f4f28b40068d47e1cc777 in term 0.
I20260927 16:31:27.354803 31942 raft_consensus.cc:2454] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2426f34ddd8f4f28b40068d47e1cc777 in term 0.
I20260927 16:31:27.355273 32011 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527; no voters:
I20260927 16:31:27.355577 32138 raft_consensus.cc:2781] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:31:27.355667 32138 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:31:27.355719 32138 raft_consensus.cc:3037] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:27.356748 32138 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.357124 32138 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [CANDIDATE]: Term 1 election: Requested vote from peers 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937), 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189)
I20260927 16:31:27.357466 31803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b75f3850d4401b60379f2c005ddb1" candidate_uuid: "2426f34ddd8f4f28b40068d47e1cc777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
I20260927 16:31:27.357466 31942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b75f3850d4401b60379f2c005ddb1" candidate_uuid: "2426f34ddd8f4f28b40068d47e1cc777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
I20260927 16:31:27.357596 31803 raft_consensus.cc:3037] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:27.357596 31942 raft_consensus.cc:3037] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:31:27.358881 31942 raft_consensus.cc:2454] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2426f34ddd8f4f28b40068d47e1cc777 in term 1.
I20260927 16:31:27.358881 31803 raft_consensus.cc:2454] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2426f34ddd8f4f28b40068d47e1cc777 in term 1.
I20260927 16:31:27.359156 32011 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527, 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters:
I20260927 16:31:27.359297 32138 raft_consensus.cc:2781] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:31:27.359536 32138 raft_consensus.cc:686] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 LEADER]: Becoming Leader. State: Replica: 2426f34ddd8f4f28b40068d47e1cc777, State: Running, Role: LEADER
I20260927 16:31:27.359920 32138 consensus_queue.cc:237] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:27.363024 31661 catalog_manager.cc:5495] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "2426f34ddd8f4f28b40068d47e1cc777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } health_report { overall_health: UNKNOWN } } }
I20260927 16:31:27.378664 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.375582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.15.93.131:40177)
0927 16:31:27.375711 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:27.375715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:27.375730 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:31:27.375878 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:27.375880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:27.375932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:27.376022 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:31:27.376215 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.376704 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.377466 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:27.377667 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:27.377740 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:27.377976 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:31:27.378372 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:31:27.378459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:27.378509 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 16:31:27.378556 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:31:27.384519 31942 raft_consensus.cc:1261] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Refusing update from remote peer 2426f34ddd8f4f28b40068d47e1cc777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:31:27.384519 31803 raft_consensus.cc:1261] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Refusing update from remote peer 2426f34ddd8f4f28b40068d47e1cc777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:31:27.385257 32138 consensus_queue.cc:1035] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:31:27.385483 32140 consensus_queue.cc:1035] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:31:27.391116 32143 mvcc.cc:205] Tried to move back new op lower bound from 7333997311518937088 to 7333997311428939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333997311518937088}]
I20260927 16:31:27.392552 32145 mvcc.cc:205] Tried to move back new op lower bound from 7333997311518937088 to 7333997311428939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333997311518937088 or (T in {7333997311518937088})}]
I20260927 16:31:27.394706 32144 mvcc.cc:205] Tried to move back new op lower bound from 7333997311518937088 to 7333997311428939776. Current Snapshot: MvccSnapshot[applied={T|T < 7333997311518937088}]
I20260927 16:31:27.399331 31659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:31:27.399515 31659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:31:27.401717 31659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Sending DeleteTablet for 3 replicas of tablet 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:27.402316 31922 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b75f3850d4401b60379f2c005ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:27 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:48344
I20260927 16:31:27.402320 31783 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b75f3850d4401b60379f2c005ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:27 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:38614
I20260927 16:31:27.402454 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31699
I20260927 16:31:27.402513 31995 tablet_replica.cc:330] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: stopping tablet replica
I20260927 16:31:27.402647 31995 raft_consensus.cc:2229] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:31:27.402773 31995 raft_consensus.cc:2258] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:27.403465 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b75f3850d4401b60379f2c005ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:31:27 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:40540
I20260927 16:31:27.403930 32154 tablet_replica.cc:330] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: stopping tablet replica
I20260927 16:31:27.404193 32154 raft_consensus.cc:2229] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:31:27.404506 32154 raft_consensus.cc:2258] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:31:27.405966 32154 ts_tablet_manager.cc:1905] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:31:27.408584 31995 ts_tablet_manager.cc:1905] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:31:27.408666 32154 ts_tablet_manager.cc:1918] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:31:27.408759 32154 log.cc:1196] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:27.409103 32154 ts_tablet_manager.cc:1939] T 660b75f3850d4401b60379f2c005ddb1 P 2426f34ddd8f4f28b40068d47e1cc777: Deleting consensus metadata
I20260927 16:31:27.410187 31646 catalog_manager.cc:4841] TS 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): tablet 660b75f3850d4401b60379f2c005ddb1 (table test-table [id=27fc8e2a77254f09904bc500cde8cf25]) successfully deleted
I20260927 16:31:27.410583 31995 ts_tablet_manager.cc:1918] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:31:27.410671 31995 log.cc:1196] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:27.410914 31995 ts_tablet_manager.cc:1939] T 660b75f3850d4401b60379f2c005ddb1 P 376aeed9ae8c4f0494c2f8fbb9d40527: Deleting consensus metadata
I20260927 16:31:27.411523 31645 catalog_manager.cc:4841] TS 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): tablet 660b75f3850d4401b60379f2c005ddb1 (table test-table [id=27fc8e2a77254f09904bc500cde8cf25]) successfully deleted
W20260927 16:31:27.412565 31646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:38189 (error 108)
I20260927 16:31:27.413021 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31853
I20260927 16:31:27.415517 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:27.415186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:38189 (local address 127.0.0.1:38618)
0927 16:31:27.415352 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:27.415439 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:38189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 16:31:27.415683 31646 catalog_manager.cc:4584] TS 63af3b5cd0e54c3c8573eb06bbfac1e0 (127.15.93.129:38189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 660b75f3850d4401b60379f2c005ddb1: Network error: Client connection negotiation failed: client connection to 127.15.93.129:38189: connect: Connection refused (error 111)
I20260927 16:31:27.419157 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31992
I20260927 16:31:27.424130 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 31628
I20260927 16:31:47.430644 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--webserver_interface=127.15.93.190
--webserver_port=38083
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:47.544116 32156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:47.544374 32156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:47.544414 32156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:47.548609 32156 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 16:31:47.548659 32156 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 16:31:47.548676 32156 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 16:31:47.548691 32156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:31:47.548745 32156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:47.548776 32156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:47.548810 32156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:31:47.548851 32156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:31:47.554136 32156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=38083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:47.555244 32156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:47.556299 32156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:47.564683 32162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:47.564761 32156 server_base.cc:1034] running on GCE node
W20260927 16:31:47.564683 32161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:47.564906 32164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:47.565562 32156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:47.566582 32156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:47.567739 32156 hybrid_clock.cc:648] HybridClock initialized: now 1790526707567705 us; error 47 us; skew 500 ppm
Sep 27 16:31:47 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:47.570571 32156 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:31:47.571365 32156 webserver.cc:460] Webserver started at http://127.15.93.190:38083/ using document root <none> and password file <none>
I20260927 16:31:47.571976 32156 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:47.572033 32156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:47.575711 32156 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 16:31:47.577893 32171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:47.579149 32156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:47.579244 32156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "1db7917f4c934e85837edcb7e357b616"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "484bbffe1f907e5a9efb162f4d7342d2"
server_key_iv: "5308c679e9af3ebda9abd451c6df6bf7"
server_key_version: "encryptionkey@0"
I20260927 16:31:47.579840 32156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:47.590472 32156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:47.593613 32156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:47.593760 32156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:47.600610 32156 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38483
I20260927 16:31:47.600898 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38483 every 8 connection(s)
I20260927 16:31:47.601678 32156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:31:47.605796 32224 sys_catalog.cc:263] Verifying existing consensus state
I20260927 16:31:47.607431 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap starting.
I20260927 16:31:47.607972 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 32156
I20260927 16:31:47.608417 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:38189
--local_ip_for_outbound_sockets=127.15.93.129
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=35233
--webserver_interface=127.15.93.129
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:31:47.631438 32224 log.cc:824] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:47.645278 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:31:47.645696 32224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Bootstrap complete.
I20260927 16:31:47.651232 32224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:47.651760 32224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Initialized, Role: FOLLOWER
I20260927 16:31:47.652319 32224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:47.652467 32224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:31:47.652525 32224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:31:47.652606 32224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 2 FOLLOWER]: Advancing to term 3
I20260927 16:31:47.655051 32224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:47.655436 32224 leader_election.cc:302] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db7917f4c934e85837edcb7e357b616; no voters:
I20260927 16:31:47.655764 32224 leader_election.cc:288] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 16:31:47.655901 32228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 16:31:47.656164 32228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [term 3 LEADER]: Becoming Leader. State: Replica: 1db7917f4c934e85837edcb7e357b616, State: Running, Role: LEADER
I20260927 16:31:47.656514 32228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } }
I20260927 16:31:47.657040 32224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:31:47.657941 32230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db7917f4c934e85837edcb7e357b616. Latest consensus state: current_term: 3 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:47.658069 32230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:47.658367 32229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1db7917f4c934e85837edcb7e357b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db7917f4c934e85837edcb7e357b616" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38483 } } }
I20260927 16:31:47.658442 32229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616 [sys.catalog]: This master's current role is: LEADER
I20260927 16:31:47.658450 32237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:31:47.663625 32237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:31:47.664383 32237 catalog_manager.cc:1259] Loaded cluster ID: 3bce9d0b91154dd6812fa037ad65360c
I20260927 16:31:47.664485 32237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:31:47.667033 32237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:31:47.667847 32237 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Loaded TSK: 0
I20260927 16:31:47.667994 32237 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1db7917f4c934e85837edcb7e357b616: Loaded TSK: 1
F20260927 16:31:47.669119 32237 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f29634329ad google::LogMessage::Fail() at ??:0
@ 0x7f296343588a google::LogMessage::SendToLog() at ??:0
@ 0x7f29634324de google::LogMessage::Flush() at ??:0
@ 0x7f2963433a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29660f618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f29660eef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f296dd977bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f296dd7199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f296dd72178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f296ddb3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f296ddd282b std::function<>::operator()() at ??:0
@ 0x7f296dd71b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f296dd72f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f296dd6f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f296ddb3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f29656843d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2965684c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f29656863f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2965675512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2966cb96db start_thread at ??:0
@ 0x7f2961f3061f clone at ??:0
W20260927 16:31:47.752178 32226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:47.752408 32226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:47.752449 32226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:47.756916 32226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:47.756985 32226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:47.757048 32226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:31:47.762987 32226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:38189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=35233
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:47.764106 32226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:47.765106 32226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:47.775049 32271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:47.775049 32272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:47.775161 32274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:47.775538 32226 server_base.cc:1034] running on GCE node
I20260927 16:31:47.775981 32226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:47.776620 32226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:47.777804 32226 hybrid_clock.cc:648] HybridClock initialized: now 1790526707777778 us; error 44 us; skew 500 ppm
Sep 27 16:31:47 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:47.780721 32226 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:31:47.781847 32226 webserver.cc:460] Webserver started at http://127.15.93.129:35233/ using document root <none> and password file <none>
I20260927 16:31:47.782604 32226 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:47.782699 32226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:47.787810 32226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:31:47.790483 32281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:47.792026 32226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:31:47.792160 32226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "bb4233e8bc7139a39f4b04414d4c5986"
server_key_iv: "a2f863851e423bd1fa87c9410d995630"
server_key_version: "encryptionkey@0"
I20260927 16:31:47.792835 32226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:47.819842 32226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:47.823730 32226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:47.823936 32226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:47.824692 32226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:47.826028 32288 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 16:31:47.830086 32226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 16:31:47.830169 32226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 16:31:47.830235 32226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 16:31:47.832247 32226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 16:31:47.832304 32226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 16:31:47.832551 32288 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap starting.
I20260927 16:31:47.843273 32226 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:38189
I20260927 16:31:47.843515 32395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:38189 every 8 connection(s)
I20260927 16:31:47.844630 32226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:31:47.845781 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 32226
I20260927 16:31:47.846325 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:34937
--local_ip_for_outbound_sockets=127.15.93.130
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=46595
--webserver_interface=127.15.93.130
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:31:47.854566 32288 log.cc:824] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:31:47.861320 32288 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 16:31:47.861712 32288 tablet_bootstrap.cc:492] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Bootstrap complete.
I20260927 16:31:47.862835 32288 ts_tablet_manager.cc:1397] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.009s
I20260927 16:31:47.865216 32288 raft_consensus.cc:348] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:47.865648 32288 raft_consensus.cc:729] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63af3b5cd0e54c3c8573eb06bbfac1e0, State: Initialized, Role: FOLLOWER
I20260927 16:31:47.866125 32288 consensus_queue.cc:260] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:47.867043 32288 ts_tablet_manager.cc:1428] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
W20260927 16:31:47.940652 32399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:31:47.848270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:41233)
0927 16:31:47.849657 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:47.849740 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:47.850848 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:47.939865 (+ 89017us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":160,"threads_started":1}
W20260927 16:31:47.941145 32396 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:38483 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:38483: Client connection negotiation failed: client connection to 127.15.93.190:38483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 16:31:47.941867 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:47.941632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:45903)
0927 16:31:47.941753 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:47.941815 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:31:47.942540 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:47.942322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:47169)
0927 16:31:47.942447 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:47.942491 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 16:31:47.942817 32396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 16:31:47.983666 32398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:47.983947 32398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:47.983994 32398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:47.988190 32398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:47.988255 32398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:47.988346 32398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:31:47.993758 32398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:34937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=46595
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:47.994858 32398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:47.996021 32398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:48.005791 32406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:48.005899 32407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:48.006079 32409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:48.006119 32398 server_base.cc:1034] running on GCE node
I20260927 16:31:48.006707 32398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:48.007354 32398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:48.008594 32398 hybrid_clock.cc:648] HybridClock initialized: now 1790526708008566 us; error 51 us; skew 500 ppm
Sep 27 16:31:48 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526708, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:48.011266 32398 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:31:48.012142 32398 webserver.cc:460] Webserver started at http://127.15.93.130:46595/ using document root <none> and password file <none>
I20260927 16:31:48.012704 32398 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:48.012795 32398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:48.016615 32398 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 16:31:48.018653 32416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.020006 32398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:48.020166 32398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "376aeed9ae8c4f0494c2f8fbb9d40527"
format_stamp: "Formatted at 2026-09-27 16:31:05 on dist-test-slave-grzf"
server_key: "e97f318cf97591e3ded8705c89b9e0e7"
server_key_iv: "e71ceeaa90c7c2c48d7983b18e67ad9f"
server_key_version: "encryptionkey@0"
I20260927 16:31:48.020715 32398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:48.045094 32398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:48.048544 32398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:48.048774 32398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:48.049393 32398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:48.050222 32398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:48.050292 32398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.050362 32398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:48.050414 32398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.059329 32398 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34937
I20260927 16:31:48.059351 32529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34937 every 8 connection(s)
I20260927 16:31:48.060354 32398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:31:48.063980 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.062407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:43917)
0927 16:31:48.062839 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.063265 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
W20260927 16:31:48.064421 32530 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:38483 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:38483: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
I20260927 16:31:48.065001 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.064784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:43873)
0927 16:31:48.064902 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.064948 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 16:31:48.065667 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.065473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:43491)
0927 16:31:48.065564 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.065620 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 16:31:48.065889 32530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:31:48.066382 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 32398
I20260927 16:31:48.066804 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:40177
--local_ip_for_outbound_sockets=127.15.93.131
--tserver_master_addrs=127.15.93.190:38483
--webserver_port=46797
--webserver_interface=127.15.93.131
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:31:48.181401 32533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:31:48.181689 32533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:31:48.181732 32533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:31:48.185870 32533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:31:48.185932 32533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:31:48.186028 32533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:31:48.191334 32533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:40177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=46797
--tserver_master_addrs=127.15.93.190:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:31:48.192440 32533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:31:48.193493 32533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:31:48.203003 32538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:48.202992 32539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:31:48.203058 32541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:31:48.203495 32533 server_base.cc:1034] running on GCE node
I20260927 16:31:48.203884 32533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:31:48.204545 32533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:31:48.205739 32533 hybrid_clock.cc:648] HybridClock initialized: now 1790526708205726 us; error 32 us; skew 500 ppm
Sep 27 16:31:48 dist-test-slave-grzf krb5kdc[31079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526708, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:31:48.208470 32533 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:31:48.209311 32533 webserver.cc:460] Webserver started at http://127.15.93.131:46797/ using document root <none> and password file <none>
I20260927 16:31:48.209873 32533 fs_manager.cc:359] Metadata directory not provided
I20260927 16:31:48.209962 32533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:31:48.213910 32533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:31:48.216135 32548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.217306 32533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:31:48.217445 32533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "2426f34ddd8f4f28b40068d47e1cc777"
format_stamp: "Formatted at 2026-09-27 16:31:06 on dist-test-slave-grzf"
server_key: "4c4303203c1b5a3a0eacde269efd8d13"
server_key_iv: "ae5d824e78c276aa0f094010df489487"
server_key_version: "encryptionkey@0"
I20260927 16:31:48.217995 32533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:31:48.253959 32533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:31:48.256992 32533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:31:48.257305 32533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:31:48.257922 32533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:31:48.258776 32533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:31:48.258847 32533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.258919 32533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:31:48.258970 32533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:31:48.268108 32533 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40177
I20260927 16:31:48.268131 32661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40177 every 8 connection(s)
I20260927 16:31:48.269117 32533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:31:48.273059 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.271453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:40507)
0927 16:31:48.271908 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.272330 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260927 16:31:48.273509 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 32533
W20260927 16:31:48.274008 32662 heartbeater.cc:644] Failed to heartbeat to 127.15.93.190:38483 (0 consecutive failures): Network error: Failed to ping master at 127.15.93.190:38483: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
I20260927 16:31:48.274703 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.274417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:47795)
0927 16:31:48.274556 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.274617 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 16:31:48.275360 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.275163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:42061)
0927 16:31:48.275270 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.275313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 16:31:48.275628 32662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 16:31:48.943917 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:48.943492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:40665)
0927 16:31:48.943759 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:48.943809 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260927 16:31:49.066871 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.066443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:42689)
0927 16:31:49.066711 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:49.066768 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260927 16:31:49.276610 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.276179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:56321)
0927 16:31:49.276434 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:49.276492 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260927 16:31:49.410015 32669 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:49.410255 32669 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:49.411511 32669 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
Sep 27 16:31:49 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.130@KRBTEST.COM
Sep 27 16:31:49 dist-test-slave-grzf krb5kdc[31079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526707, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.131@KRBTEST.COM
I20260927 16:31:49.428546 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.411733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34937 (local address 127.15.93.129:54977)
0927 16:31:49.412084 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:49.412098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:49.412199 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:49.414232 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:49.414238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:49.414857 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:49.415673 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:49.415695 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.417631 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.417635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:49.419340 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:49.419353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.420244 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.420248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:49.420349 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:49.420975 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:49.421000 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:49.422868 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:49.425348 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:49.425355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:49.425374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:49.425808 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:49.426301 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:49.426306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:49.426524 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:49.427291 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:49.427298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:49.427681 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 16:31:49.427919 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 16:31:49.428390 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":440,"spinlock_wait_cycles":12288,"thread_start_us":93,"threads_started":1}
I20260927 16:31:49.428583 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.411870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:53881 (local address 127.15.93.131:40177)
0927 16:31:49.412276 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:49.412303 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:49.412347 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:31:49.413272 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:49.413281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:49.413757 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:49.414107 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:49.415870 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.417455 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.419498 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.420004 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.420135 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:49.423451 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:49.423456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:49.423468 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:49.423498 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:49.425419 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426058 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:49.426063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:49.426069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:49.426195 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426635 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:49.426638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:49.426640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:49.427165 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:49.427304 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:49.428055 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 16:31:49.428357 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":91,"threads_started":1}
I20260927 16:31:49.428545 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.411712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40177 (local address 127.15.93.129:53881)
0927 16:31:49.411916 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:49.411932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:31:49.412036 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:31:49.414232 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:31:49.414238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:31:49.414949 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:31:49.415673 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:49.415695 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.417631 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.417635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:49.419340 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:31:49.419353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.420143 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.420150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:31:49.420327 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:49.421114 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:31:49.421125 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:31:49.423216 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:31:49.425541 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:49.425544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:49.425546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:49.425877 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:49.426301 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:31:49.426306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:31:49.426524 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:31:49.427291 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:31:49.427298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:31:49.427681 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 16:31:49.427919 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 16:31:49.428339 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":355,"spinlock_wait_cycles":23168}
I20260927 16:31:49.428581 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.411990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:54977 (local address 127.15.93.130:34937)
0927 16:31:49.412302 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 16:31:49.412321 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:31:49.412352 (+ 31us) server_negotiation.cc:407] Connection header received
0927 16:31:49.413272 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:31:49.413281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:31:49.413757 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:31:49.414107 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:31:49.415870 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.417455 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.419526 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:31:49.420108 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:31:49.420271 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:31:49.423155 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:31:49.423161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:31:49.423173 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:31:49.423203 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:31:49.425214 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426017 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:49.426025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:49.426031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:49.426195 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:31:49.426635 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:31:49.426638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:31:49.426640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:31:49.427178 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:31:49.427304 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 16:31:49.428055 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 16:31:49.428357 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
W20260927 16:31:49.431300 32283 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
W20260927 16:31:49.431532 32282 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:49.431587 32283 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:49.431833 32669 raft_consensus.cc:2726] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:49.945155 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:49.944719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:36735)
0927 16:31:49.944979 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:49.945058 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 16:31:50.068149 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:50.067729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:53899)
0927 16:31:50.067976 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:50.068044 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260927 16:31:50.277896 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:50.277370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:58607)
0927 16:31:50.277684 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:50.277766 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260927 16:31:50.946560 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:50.946086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:53477)
0927 16:31:50.946397 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:50.946457 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":117,"threads_started":1}
I20260927 16:31:51.069376 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:51.068942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:48283)
0927 16:31:51.069208 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:51.069266 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260927 16:31:51.279110 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:51.278666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:38725)
0927 16:31:51.278936 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:51.278995 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260927 16:31:51.584947 32679 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:51.585081 32679 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:51.585570 32679 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
W20260927 16:31:51.586488 32282 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
W20260927 16:31:51.586614 32283 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:51.586788 32283 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:51.587073 32679 raft_consensus.cc:2726] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:51.947868 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:51.947433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:45483)
0927 16:31:51.947708 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:51.947772 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260927 16:31:52.070710 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:52.070235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:37793)
0927 16:31:52.070530 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:52.070618 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260927 16:31:52.280210 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:52.279824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:38917)
0927 16:31:52.280050 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:52.280115 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260927 16:31:52.949270 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:52.948716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:46917)
0927 16:31:52.949014 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:52.949119 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":71,"threads_started":1}
I20260927 16:31:53.072190 32684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:53.071686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:57811)
0927 16:31:53.072028 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:53.072091 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260927 16:31:53.103995 32685 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:53.104182 32685 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:53.104780 32685 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
W20260927 16:31:53.105561 32282 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
W20260927 16:31:53.105654 32283 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:53.105710 32283 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:53.105858 32685 raft_consensus.cc:2726] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:53.281469 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:53.280993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:39399)
0927 16:31:53.281309 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:53.281375 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260927 16:31:53.950534 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:53.950060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:35493)
0927 16:31:53.950367 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:53.950435 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
I20260927 16:31:54.073527 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:54.073020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:52317)
0927 16:31:54.073331 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:54.073396 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260927 16:31:54.282807 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:54.282291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:47301)
0927 16:31:54.282645 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:54.282709 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":73,"threads_started":1}
I20260927 16:31:54.951713 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:54.951279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:57685)
0927 16:31:54.951552 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:54.951619 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":112,"threads_started":1}
I20260927 16:31:54.976137 32691 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:54.976312 32691 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:54.976776 32691 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
W20260927 16:31:54.977514 32283 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
W20260927 16:31:54.977648 32282 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:54.977737 32282 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:54.977887 32691 raft_consensus.cc:2726] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:55.074869 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:55.074351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:36145)
0927 16:31:55.074705 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:55.074771 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
I20260927 16:31:55.284178 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:55.283661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:35853)
0927 16:31:55.284012 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:55.284072 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":73,"threads_started":1}
I20260927 16:31:55.952950 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:55.952480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:34859)
0927 16:31:55.952775 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:55.952837 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260927 16:31:56.076241 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:56.075788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:50591)
0927 16:31:56.076066 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:56.076133 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260927 16:31:56.285401 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:56.284942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:52669)
0927 16:31:56.285228 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:56.285290 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260927 16:31:56.954206 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:56.953759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:38499)
0927 16:31:56.954046 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:56.954108 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260927 16:31:57.077548 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:57.077063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:56849)
0927 16:31:57.077372 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:57.077441 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260927 16:31:57.286931 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:57.286354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:60079)
0927 16:31:57.286720 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:57.286802 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260927 16:31:57.955683 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:57.955233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:42303)
0927 16:31:57.955519 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:57.955580 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260927 16:31:58.078994 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:58.078493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:58473)
0927 16:31:58.078824 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:58.078893 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":116,"threads_started":1}
I20260927 16:31:58.288343 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:58.287884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:36849)
0927 16:31:58.288182 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:58.288247 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260927 16:31:58.956862 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:58.956447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:37047)
0927 16:31:58.956702 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:58.956756 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260927 16:31:59.080345 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:59.079900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:56295)
0927 16:31:59.080188 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:59.080249 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":94,"threads_started":1}
I20260927 16:31:59.289703 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:59.289188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:40779)
0927 16:31:59.289523 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:59.289605 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
I20260927 16:31:59.313265 32706 raft_consensus.cc:482] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:31:59.313422 32706 raft_consensus.cc:504] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2426f34ddd8f4f28b40068d47e1cc777" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40177 } } peers { permanent_uuid: "376aeed9ae8c4f0494c2f8fbb9d40527" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34937 } } peers { permanent_uuid: "63af3b5cd0e54c3c8573eb06bbfac1e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 38189 } }
I20260927 16:31:59.313905 32706 leader_election.cc:288] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177), 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937)
W20260927 16:31:59.314730 32282 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 376aeed9ae8c4f0494c2f8fbb9d40527 (127.15.93.130:34937): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
W20260927 16:31:59.314834 32283 leader_election.cc:341] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2426f34ddd8f4f28b40068d47e1cc777 (127.15.93.131:40177): Not found: Tablet not found: 660b75f3850d4401b60379f2c005ddb1
I20260927 16:31:59.314955 32283 leader_election.cc:302] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63af3b5cd0e54c3c8573eb06bbfac1e0; no voters: 2426f34ddd8f4f28b40068d47e1cc777, 376aeed9ae8c4f0494c2f8fbb9d40527
I20260927 16:31:59.315128 32706 raft_consensus.cc:2726] T 660b75f3850d4401b60379f2c005ddb1 P 63af3b5cd0e54c3c8573eb06bbfac1e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 16:31:59.958122 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:31:59.957660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:47889)
0927 16:31:59.957944 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 16:31:59.958011 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260927 16:32:00.081493 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:00.081083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:52969)
0927 16:32:00.081340 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:00.081397 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260927 16:32:00.290874 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:00.290448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:53919)
0927 16:32:00.290727 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:00.290780 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260927 16:32:00.959470 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:00.958948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:41091)
0927 16:32:00.959227 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:00.959291 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260927 16:32:01.082857 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:01.082330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:48391)
0927 16:32:01.082641 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:01.082704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260927 16:32:01.291990 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:01.291569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:40649)
0927 16:32:01.291841 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:01.291896 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260927 16:32:01.960709 32713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:01.960292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:53563)
0927 16:32:01.960538 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:01.960613 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260927 16:32:02.084241 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:02.083815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:52449)
0927 16:32:02.084082 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:02.084143 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260927 16:32:02.293378 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:02.292792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.131:50907)
0927 16:32:02.293146 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:02.293217 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260927 16:32:02.962023 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:02.961646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.129:60265)
0927 16:32:02.961875 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:02.961928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260927 16:32:03.085676 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:03.085254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38483 (local address 127.15.93.130:57363)
0927 16:32:03.085516 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:03.085582 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.190:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260927 16:32:03.276772 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 32226
I20260927 16:32:03.282785 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 32398
I20260927 16:32:03.288064 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 32533
2026-09-27T16:32:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59205 ms)
[----------] 30 tests from SecurityITest (235146 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:03 dist-test-slave-grzf krb5kdc[32720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:03 dist-test-slave-grzf krb5kdc[32720](info): set up 2 sockets
Sep 27 16:32:03 dist-test-slave-grzf krb5kdc[32720](info): commencing operation
krb5kdc: starting...
W20260927 16:32:04.335201 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:04 dist-test-slave-grzf krb5kdc[32720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:04.452243 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:37057
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:37057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:04.559201 32733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:04.559495 32733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:04.559583 32733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:04.563563 32733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:04.563625 32733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:04.563644 32733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:04.563663 32733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:04.563722 32733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:04.568941 32733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:37057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:37057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:04.570181 32733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:04.571645 32733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:04.580550 32741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:04.580691 32739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:04.580627 32733 server_base.cc:1034] running on GCE node
W20260927 16:32:04.580598 32738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:04.581453 32733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:04.582520 32733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:04.583709 32733 hybrid_clock.cc:648] HybridClock initialized: now 1790526724583679 us; error 40 us; skew 500 ppm
Sep 27 16:32:04 dist-test-slave-grzf krb5kdc[32720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526724, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:04.586584 32733 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:04.587587 32733 webserver.cc:460] Webserver started at http://127.15.93.190:37677/ using document root <none> and password file <none>
I20260927 16:32:04.588158 32733 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:04.588245 32733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:04.588495 32733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:04.590521 32733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "96babb2694de475295e519d4a3e62cce"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "124b4837e5df58db145004d1ec809389"
server_key_iv: "77460a9a6c36df3527d7baa3c926c0f8"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.591090 32733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "96babb2694de475295e519d4a3e62cce"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "124b4837e5df58db145004d1ec809389"
server_key_iv: "77460a9a6c36df3527d7baa3c926c0f8"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.594749 32733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:32:04.597101 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:04.598352 32733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:32:04.598490 32733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "96babb2694de475295e519d4a3e62cce"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "124b4837e5df58db145004d1ec809389"
server_key_iv: "77460a9a6c36df3527d7baa3c926c0f8"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.598600 32733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:04.615000 32733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:04.618242 32733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:04.618445 32733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:04.625234 32733 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:37057
I20260927 16:32:04.625262 332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:37057 every 8 connection(s)
I20260927 16:32:04.626127 32733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:04.627803 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 32733
I20260927 16:32:04.627940 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:04.628278 15734 external_mini_cluster.cc:1428] Setting key 3861621dcff572f13e7a2efbc6aab9a3
I20260927 16:32:04.629005 333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:04.634510 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: Bootstrap starting.
Sep 27 16:32:04 dist-test-slave-grzf krb5kdc[32720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:04.637176 333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:04.638060 333 log.cc:824] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:04.640232 333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: No bootstrap required, opened a new log
I20260927 16:32:04.642331 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:04.629615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.15.93.190:37057)
0927 16:32:04.630076 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:04.630091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:04.630130 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:32:04.630800 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:04.630817 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:04.631135 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:04.631521 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:04.632290 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.633256 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.634113 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.634584 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.634763 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:04.637490 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:04.637496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:04.637510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:04.637539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:04.639320 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:04.639768 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:04.639774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:04.639779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:04.639908 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:04.640213 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:04.640216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:04.640218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:04.640713 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:04.640892 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:04.641146 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 16:32:04.641416 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260927 16:32:04.644440 333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } }
I20260927 16:32:04.644694 333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:04.644762 333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96babb2694de475295e519d4a3e62cce, State: Initialized, Role: FOLLOWER
I20260927 16:32:04.645205 333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } }
I20260927 16:32:04.645347 333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:04.645408 333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:04.645501 333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:04.646348 333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } }
I20260927 16:32:04.646682 333 leader_election.cc:302] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96babb2694de475295e519d4a3e62cce; no voters:
I20260927 16:32:04.646963 333 leader_election.cc:288] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:04.647101 338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:04.647333 338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [term 1 LEADER]: Becoming Leader. State: Replica: 96babb2694de475295e519d4a3e62cce, State: Running, Role: LEADER
I20260927 16:32:04.647881 338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } }
I20260927 16:32:04.648228 333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:04.648916 338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96babb2694de475295e519d4a3e62cce. Latest consensus state: current_term: 1 leader_uuid: "96babb2694de475295e519d4a3e62cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } } }
I20260927 16:32:04.649122 338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:04.649326 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96babb2694de475295e519d4a3e62cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96babb2694de475295e519d4a3e62cce" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37057 } } }
I20260927 16:32:04.649446 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:04.649621 347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:04.652647 347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:04.658149 347 catalog_manager.cc:1347] Generated new cluster ID: 7f8c7dc49d864fc0adf09e09d280eb80
I20260927 16:32:04.658252 347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:04.676896 347 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:04.677843 347 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:04.688727 347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: Generated new TSK 0
I20260927 16:32:04.689492 347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:32:04.716699 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:04.827589 359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:04.827842 359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:04.827919 359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:04.831938 359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:04.831995 359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:04.832051 359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:04.837206 359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:04.838258 359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:04.839275 359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:04.848551 364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:04.848552 365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:04.848552 367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:04.848868 359 server_base.cc:1034] running on GCE node
I20260927 16:32:04.849346 359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:04.850037 359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:04.851190 359 hybrid_clock.cc:648] HybridClock initialized: now 1790526724851170 us; error 32 us; skew 500 ppm
Sep 27 16:32:04 dist-test-slave-grzf krb5kdc[32720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526724, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:04.854117 359 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:32:04.855012 359 webserver.cc:460] Webserver started at http://127.15.93.129:45449/ using document root <none> and password file <none>
I20260927 16:32:04.855677 359 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:04.855769 359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:04.855996 359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:04.858021 359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "f0269751dd7f410eac965f23506bce36"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "eb71e286d70a46382b1caf9a0784146e"
server_key_iv: "e1530363eb2f52fe16dd6eb2cbdce0c5"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.858589 359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "f0269751dd7f410eac965f23506bce36"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "eb71e286d70a46382b1caf9a0784146e"
server_key_iv: "e1530363eb2f52fe16dd6eb2cbdce0c5"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.862293 359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:32:04.864697 374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:04.865875 359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:04.866014 359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "f0269751dd7f410eac965f23506bce36"
format_stamp: "Formatted at 2026-09-27 16:32:04 on dist-test-slave-grzf"
server_key: "eb71e286d70a46382b1caf9a0784146e"
server_key_iv: "e1530363eb2f52fe16dd6eb2cbdce0c5"
server_key_version: "encryptionkey@0"
I20260927 16:32:04.866129 359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:04.890282 359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:04.893602 359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:04.893797 359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:04.894454 359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:04.895287 359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:04.895355 359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:04.895468 359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:04.895524 359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:04.904664 359 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:32905
I20260927 16:32:04.904686 487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:32905 every 8 connection(s)
I20260927 16:32:04.905583 359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:04.913434 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 359
I20260927 16:32:04.913580 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:04.913939 15734 external_mini_cluster.cc:1428] Setting key c15bc8acfd206c12013685b02dae3e44
Sep 27 16:32:04 dist-test-slave-grzf krb5kdc[32720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526724, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:04.919561 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:04.907452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:40263 (local address 127.15.93.190:37057)
0927 16:32:04.907587 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:04.907591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:04.908406 (+ 815us) server_negotiation.cc:407] Connection header received
0927 16:32:04.909737 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:04.909741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:04.909797 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:04.909873 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:04.911147 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.911770 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.912857 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.913072 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.913149 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:04.915932 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:04.915935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:04.915937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:04.915959 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:04.917553 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:04.918027 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:04.918031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:04.918033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:04.918135 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:04.918457 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:04.918462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:04.918464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:04.918645 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:04.918751 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:04.919223 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 16:32:04.919343 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:32:04.920331 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:04.907825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37057 (local address 127.15.93.129:40263)
0927 16:32:04.908256 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:04.908299 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:04.909514 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:04.910061 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:04.910068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:04.910431 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:04.910969 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:04.910993 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.911877 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.911880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:04.912752 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:04.912761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:04.913213 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:04.913216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:04.913349 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:04.913957 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:04.913992 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:04.915775 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:04.917671 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:04.917675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:04.917687 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:04.917930 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:04.918229 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:04.918232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:04.918233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:04.918355 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:04.918760 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:04.918769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:04.918997 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 16:32:04.919184 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 16:32:04.919522 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260927 16:32:04.921736 488 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37057
I20260927 16:32:04.921967 488 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:04.922429 488 heartbeater.cc:511] Master 127.15.93.190:37057 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:32:04.924098 32765 ts_manager.cc:194] Registered new tserver with Master: f0269751dd7f410eac965f23506bce36 (127.15.93.129:32905)
I20260927 16:32:04.925276 32765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:40263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:32:04.942766 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:05.056167 493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:05.056427 493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:05.056515 493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:05.060609 493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:05.060665 493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:05.060770 493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:05.066040 493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:05.067153 493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:05.068239 493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:05.077932 501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.078053 493 server_base.cc:1034] running on GCE node
W20260927 16:32:05.077972 498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:05.078233 499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.078707 493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:05.079336 493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:05.080579 493 hybrid_clock.cc:648] HybridClock initialized: now 1790526725080547 us; error 43 us; skew 500 ppm
Sep 27 16:32:05 dist-test-slave-grzf krb5kdc[32720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526725, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:05.083313 493 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:32:05.084213 493 webserver.cc:460] Webserver started at http://127.15.93.130:38755/ using document root <none> and password file <none>
I20260927 16:32:05.084784 493 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:05.084873 493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:05.085107 493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:05.087136 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "059c9dc3ec5947f380b0095ff1bf566d"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "5c285f424d30d52af197270335872391"
server_key_iv: "9eea5c1049d45e81ba57e4b1a31aca49"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.087764 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "059c9dc3ec5947f380b0095ff1bf566d"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "5c285f424d30d52af197270335872391"
server_key_iv: "9eea5c1049d45e81ba57e4b1a31aca49"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.091472 493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:32:05.093891 508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.095098 493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:05.095238 493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "059c9dc3ec5947f380b0095ff1bf566d"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "5c285f424d30d52af197270335872391"
server_key_iv: "9eea5c1049d45e81ba57e4b1a31aca49"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.095347 493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:05.113730 493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:05.116709 493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:05.116922 493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:05.117507 493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:05.118315 493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:05.118383 493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.118455 493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:05.118497 493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.126963 493 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:46007
I20260927 16:32:05.126989 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:46007 every 8 connection(s)
I20260927 16:32:05.127839 493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:05.129436 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 493
I20260927 16:32:05.129573 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:05.129948 15734 external_mini_cluster.cc:1428] Setting key 76027568671aff00dbbd0d291fad09bb
Sep 27 16:32:05 dist-test-slave-grzf krb5kdc[32720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526725, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
I20260927 16:32:05.141867 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.129630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:38773 (local address 127.15.93.190:37057)
0927 16:32:05.129787 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.129793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.131212 (+ 1419us) server_negotiation.cc:407] Connection header received
0927 16:32:05.132356 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.132359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.132411 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.132493 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:05.133648 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.134173 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.135317 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.135612 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.135720 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.138117 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:05.138120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:05.138122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:05.138144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:05.139707 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.140274 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.140278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.140280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.140361 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.140710 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.140713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.140715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.140892 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:05.141034 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.141575 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.141699 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:32:05.142650 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.130006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37057 (local address 127.15.93.130:38773)
0927 16:32:05.130508 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.130560 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.132152 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.132682 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.132689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.133041 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:05.133511 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.133523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.134295 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.134298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.135194 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.135203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.135755 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.135760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.135929 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.136569 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:05.136585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:05.137951 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:05.139836 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:05.139841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:05.139856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:05.140185 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:05.140453 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:05.140457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:05.140459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:05.140607 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:05.140994 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:05.141000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:05.141291 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.141476 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.141812 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260927 16:32:05.145128 622 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37057
I20260927 16:32:05.145392 622 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:05.146019 622 heartbeater.cc:511] Master 127.15.93.190:37057 requested a full tablet report, sending...
I20260927 16:32:05.147264 32765 ts_manager.cc:194] Registered new tserver with Master: 059c9dc3ec5947f380b0095ff1bf566d (127.15.93.130:46007)
I20260927 16:32:05.147917 32765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:38773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:32:05.158800 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:05.268927 627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:05.269212 627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:05.269316 627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:05.273377 627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:05.273432 627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:05.273488 627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:05.279090 627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:05.280198 627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:05.281126 627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:05.290586 633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.290752 627 server_base.cc:1034] running on GCE node
W20260927 16:32:05.290606 635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:05.290654 632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.291321 627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:05.291966 627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:05.293124 627 hybrid_clock.cc:648] HybridClock initialized: now 1790526725293092 us; error 45 us; skew 500 ppm
Sep 27 16:32:05 dist-test-slave-grzf krb5kdc[32720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526725, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:05.295924 627 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:32:05.296824 627 webserver.cc:460] Webserver started at http://127.15.93.131:39737/ using document root <none> and password file <none>
I20260927 16:32:05.297374 627 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:05.297465 627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:05.297683 627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:05.299870 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "e02f42c8dadb4647bb2067dd2c5cc732"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "082417db019d51f151eccefa35da06b0"
server_key_iv: "d49403095a7dcf3e71f9b67ee03e1e0b"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.300422 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "e02f42c8dadb4647bb2067dd2c5cc732"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "082417db019d51f151eccefa35da06b0"
server_key_iv: "d49403095a7dcf3e71f9b67ee03e1e0b"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.304188 627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:32:05.306569 642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.307794 627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:05.307930 627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "e02f42c8dadb4647bb2067dd2c5cc732"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "082417db019d51f151eccefa35da06b0"
server_key_iv: "d49403095a7dcf3e71f9b67ee03e1e0b"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.308037 627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:05.353369 627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:05.356909 627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:05.357126 627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:05.357738 627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:05.358578 627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:05.358657 627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.358728 627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:05.358778 627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.367841 627 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:46127
I20260927 16:32:05.367880 755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:46127 every 8 connection(s)
I20260927 16:32:05.368774 627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:32:05.375692 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 627
I20260927 16:32:05.375839 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:05.376123 15734 external_mini_cluster.cc:1428] Setting key 220e3df12bb77bdb7bc6e4d01ff02c9a
Sep 27 16:32:05 dist-test-slave-grzf krb5kdc[32720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526725, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:05.382210 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.370597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:37329 (local address 127.15.93.190:37057)
0927 16:32:05.370733 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.370738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.371665 (+ 927us) server_negotiation.cc:407] Connection header received
0927 16:32:05.373023 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.373027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.373083 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.373190 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:05.374327 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.374867 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.376185 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.376411 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.376531 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.379189 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:05.379193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:05.379194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:05.379217 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:05.380504 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.380950 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.380952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.380954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.381025 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.381306 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.381308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.381310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.381465 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:05.381562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.382004 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.382095 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:05.382988 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.371057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37057 (local address 127.15.93.131:37329)
0927 16:32:05.371513 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.371555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.372803 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.373362 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.373369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.373706 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:05.374176 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.374189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.375006 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.375013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.376060 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.376071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.376568 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.376572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.376772 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.377465 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:05.377489 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:05.379044 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:05.380612 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:05.380617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:05.380628 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:05.380869 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:05.381104 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:05.381107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:05.381109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:05.381232 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:05.381572 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:05.381577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:05.381809 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.381958 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.382194 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260927 16:32:05.384101 756 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37057
I20260927 16:32:05.384303 756 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:05.384720 756 heartbeater.cc:511] Master 127.15.93.190:37057 requested a full tablet report, sending...
I20260927 16:32:05.385674 32765 ts_manager.cc:194] Registered new tserver with Master: e02f42c8dadb4647bb2067dd2c5cc732 (127.15.93.131:46127)
I20260927 16:32:05.386236 32765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:37329
I20260927 16:32:05.389793 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:05.398797 336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.391770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.15.93.190:37057)
0927 16:32:05.391888 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.391891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.392126 (+ 235us) server_negotiation.cc:407] Connection header received
0927 16:32:05.392210 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.392212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.392247 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.392310 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:05.393104 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.393600 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.394432 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.394643 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.394723 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.396001 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:05.396004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:05.396006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:05.396027 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:05.397295 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.397695 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.397697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.397698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.397764 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:05.397982 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:05.397984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:05.397985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:05.398180 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:05.398279 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.398604 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.398689 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:32:05.401626 32765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:05.404673 32765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:05.419225 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.413062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:46127 (local address 127.0.0.1:45908)
0927 16:32:05.413734 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.413784 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.413917 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.414918 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.414920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.414934 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:05.415023 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.415031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416597 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.416600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.417871 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.417880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.418704 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.418815 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.418849 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.418957 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.419070 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":41,"spinlock_wait_cycles":6272,"thread_start_us":117,"threads_started":1}
I20260927 16:32:05.419229 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.414114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.15.93.131:46127)
0927 16:32:05.414474 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.414486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.414509 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:32:05.414580 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.414588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.414769 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.415161 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:05.415295 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416410 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418001 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.418572 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418704 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.418846 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.418995 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.419080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":132,"threads_started":1}
I20260927 16:32:05.419591 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.412967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32905 (local address 127.0.0.1:35778)
0927 16:32:05.413734 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.413784 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.413917 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.414755 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.414761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.414785 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:05.414933 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.414940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416450 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.416455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.417907 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.417915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.419059 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.419062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.419205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.419244 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.419312 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.419428 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":170,"spinlock_wait_cycles":3328,"thread_start_us":77,"threads_started":1}
I20260927 16:32:05.419651 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.413746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.15.93.130:46007)
0927 16:32:05.414680 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.414689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.415022 (+ 333us) server_negotiation.cc:407] Connection header received
0927 16:32:05.415101 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.415106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.415277 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.415476 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:05.415673 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416409 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.417872 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.418723 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.418987 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.419425 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.419516 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":145,"threads_started":1}
I20260927 16:32:05.420254 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.413100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.15.93.129:32905)
0927 16:32:05.413494 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.413509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.413897 (+ 388us) server_negotiation.cc:407] Connection header received
0927 16:32:05.414098 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.414105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.414318 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.414500 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:05.415086 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416306 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418035 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.418804 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.418870 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.418975 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.420033 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.420127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":78,"threads_started":1}
I20260927 16:32:05.420578 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.413348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:46007 (local address 127.0.0.1:53360)
0927 16:32:05.414825 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.414838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.414927 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.415452 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.415455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.415472 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:05.415561 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.415568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.416551 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.416557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.417742 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.417752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.419052 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.419056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.419209 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.419253 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.419312 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.420451 (+ 1139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":1221,"thread_start_us":83,"threads_started":1}
I20260927 16:32:05.422693 690 tablet_service.cc:1467] Processing CreateTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea (DEFAULT_TABLE table=test-table [id=bc5a839941e14ba785f8d4afd4da7681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:05.422943 422 tablet_service.cc:1467] Processing CreateTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea (DEFAULT_TABLE table=test-table [id=bc5a839941e14ba785f8d4afd4da7681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:05.423360 556 tablet_service.cc:1467] Processing CreateTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea (DEFAULT_TABLE table=test-table [id=bc5a839941e14ba785f8d4afd4da7681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:05.423844 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6635c6d9e1b4335a3ee000c539f16ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:05.423870 422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6635c6d9e1b4335a3ee000c539f16ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:05.424253 556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6635c6d9e1b4335a3ee000c539f16ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:05.430330 773 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: Bootstrap starting.
I20260927 16:32:05.430922 774 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Bootstrap starting.
I20260927 16:32:05.432647 773 tablet_bootstrap.cc:654] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:05.433318 774 tablet_bootstrap.cc:654] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:05.433504 773 log.cc:824] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:05.431310 775 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Bootstrap starting.
I20260927 16:32:05.434616 774 log.cc:824] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:05.435649 773 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: No bootstrap required, opened a new log
I20260927 16:32:05.435912 773 ts_tablet_manager.cc:1397] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:32:05.436007 775 tablet_bootstrap.cc:654] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:05.436672 774 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: No bootstrap required, opened a new log
I20260927 16:32:05.436847 774 ts_tablet_manager.cc:1397] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:05.437258 775 log.cc:824] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:05.439095 775 tablet_bootstrap.cc:492] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: No bootstrap required, opened a new log
I20260927 16:32:05.439263 775 ts_tablet_manager.cc:1397] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 16:32:05.439543 773 raft_consensus.cc:348] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.439786 773 raft_consensus.cc:374] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:05.439857 773 raft_consensus.cc:729] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0269751dd7f410eac965f23506bce36, State: Initialized, Role: FOLLOWER
I20260927 16:32:05.440316 773 consensus_queue.cc:260] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.440402 774 raft_consensus.cc:348] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.440707 774 raft_consensus.cc:374] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:05.440793 774 raft_consensus.cc:729] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02f42c8dadb4647bb2067dd2c5cc732, State: Initialized, Role: FOLLOWER
I20260927 16:32:05.440975 488 heartbeater.cc:503] Master 127.15.93.190:37057 was elected leader, sending a full tablet report...
I20260927 16:32:05.441267 773 ts_tablet_manager.cc:1428] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:32:05.441219 774 consensus_queue.cc:260] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.441816 756 heartbeater.cc:503] Master 127.15.93.190:37057 was elected leader, sending a full tablet report...
I20260927 16:32:05.442137 774 ts_tablet_manager.cc:1428] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:32:05.444559 775 raft_consensus.cc:348] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.444814 775 raft_consensus.cc:374] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:05.444911 775 raft_consensus.cc:729] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059c9dc3ec5947f380b0095ff1bf566d, State: Initialized, Role: FOLLOWER
I20260927 16:32:05.445400 775 consensus_queue.cc:260] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.446094 622 heartbeater.cc:503] Master 127.15.93.190:37057 was elected leader, sending a full tablet report...
I20260927 16:32:05.446399 775 ts_tablet_manager.cc:1428] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 16:32:05.458501 781 raft_consensus.cc:482] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:05.458739 781 raft_consensus.cc:504] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.459945 781 leader_election.cc:288] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0269751dd7f410eac965f23506bce36 (127.15.93.129:32905), e02f42c8dadb4647bb2067dd2c5cc732 (127.15.93.131:46127)
I20260927 16:32:05.464097 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.460232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32905 (local address 127.15.93.130:37377)
0927 16:32:05.460335 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.460350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.460468 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.460999 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.461003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.461017 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:05.461163 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.461170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.462023 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.462026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.462918 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.462927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.463641 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.463750 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.463777 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.463845 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.463924 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":19}
I20260927 16:32:05.464099 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.460511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37377 (local address 127.15.93.129:32905)
0927 16:32:05.460663 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.460668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.460683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:32:05.460745 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.460751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.460807 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.460882 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:05.461302 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.461914 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463048 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.463523 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463646 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.463705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.463886 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.463971 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:32:05.464139 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.460350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:60627 (local address 127.15.93.131:46127)
0927 16:32:05.460484 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.460488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.460779 (+ 291us) server_negotiation.cc:407] Connection header received
0927 16:32:05.461013 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.461017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.461077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.461182 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:05.461518 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.462079 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463102 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.463558 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463693 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.463743 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.463952 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.464036 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:32:05.464147 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.460232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:46127 (local address 127.15.93.130:60627)
0927 16:32:05.460682 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.460697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:05.460844 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:05.461198 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:05.461201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:05.461220 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:05.461376 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.461385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.462186 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.462189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.462957 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:05.462965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.463702 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.463705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:05.463808 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.463845 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:32:05.463924 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:32:05.463986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":111,"thread_start_us":121,"threads_started":1}
I20260927 16:32:05.464830 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6635c6d9e1b4335a3ee000c539f16ea" candidate_uuid: "059c9dc3ec5947f380b0095ff1bf566d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" is_pre_election: true
I20260927 16:32:05.464831 442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6635c6d9e1b4335a3ee000c539f16ea" candidate_uuid: "059c9dc3ec5947f380b0095ff1bf566d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0269751dd7f410eac965f23506bce36" is_pre_election: true
I20260927 16:32:05.465139 442 raft_consensus.cc:2454] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059c9dc3ec5947f380b0095ff1bf566d in term 0.
I20260927 16:32:05.465147 710 raft_consensus.cc:2454] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059c9dc3ec5947f380b0095ff1bf566d in term 0.
I20260927 16:32:05.465646 509 leader_election.cc:302] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059c9dc3ec5947f380b0095ff1bf566d, e02f42c8dadb4647bb2067dd2c5cc732, f0269751dd7f410eac965f23506bce36; no voters:
I20260927 16:32:05.465906 781 raft_consensus.cc:2781] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:05.465996 781 raft_consensus.cc:482] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:05.466059 781 raft_consensus.cc:3037] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:05.467072 781 raft_consensus.cc:504] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.467478 781 leader_election.cc:288] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [CANDIDATE]: Term 1 election: Requested vote from peers f0269751dd7f410eac965f23506bce36 (127.15.93.129:32905), e02f42c8dadb4647bb2067dd2c5cc732 (127.15.93.131:46127)
I20260927 16:32:05.467823 442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6635c6d9e1b4335a3ee000c539f16ea" candidate_uuid: "059c9dc3ec5947f380b0095ff1bf566d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0269751dd7f410eac965f23506bce36"
I20260927 16:32:05.467831 710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6635c6d9e1b4335a3ee000c539f16ea" candidate_uuid: "059c9dc3ec5947f380b0095ff1bf566d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02f42c8dadb4647bb2067dd2c5cc732"
I20260927 16:32:05.467957 710 raft_consensus.cc:3037] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:05.467954 442 raft_consensus.cc:3037] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:05.469002 710 raft_consensus.cc:2454] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059c9dc3ec5947f380b0095ff1bf566d in term 1.
I20260927 16:32:05.469301 509 leader_election.cc:302] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059c9dc3ec5947f380b0095ff1bf566d, e02f42c8dadb4647bb2067dd2c5cc732; no voters:
I20260927 16:32:05.469338 442 raft_consensus.cc:2454] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059c9dc3ec5947f380b0095ff1bf566d in term 1.
I20260927 16:32:05.469491 781 raft_consensus.cc:2781] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:05.469726 781 raft_consensus.cc:686] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 1 LEADER]: Becoming Leader. State: Replica: 059c9dc3ec5947f380b0095ff1bf566d, State: Running, Role: LEADER
I20260927 16:32:05.470049 781 consensus_queue.cc:237] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } }
I20260927 16:32:05.473249 32765 catalog_manager.cc:5495] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d reported cstate change: term changed from 0 to 1, leader changed from <none> to 059c9dc3ec5947f380b0095ff1bf566d (127.15.93.130). New cstate: current_term: 1 leader_uuid: "059c9dc3ec5947f380b0095ff1bf566d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059c9dc3ec5947f380b0095ff1bf566d" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46007 } health_report { overall_health: HEALTHY } } }
I20260927 16:32:05.497742 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.494503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.15.93.130:46007)
0927 16:32:05.494633 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.494637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.494652 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:32:05.494726 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.494729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.494784 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.494873 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:05.495098 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.495652 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.496468 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.496668 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.496740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.496988 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:05.497405 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:05.497493 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.497564 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.497612 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:05.503926 710 raft_consensus.cc:1261] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 1 FOLLOWER]: Refusing update from remote peer 059c9dc3ec5947f380b0095ff1bf566d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:05.503926 442 raft_consensus.cc:1261] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 1 FOLLOWER]: Refusing update from remote peer 059c9dc3ec5947f380b0095ff1bf566d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:05.504609 781 consensus_queue.cc:1035] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0269751dd7f410eac965f23506bce36" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:05.504774 783 consensus_queue.cc:1035] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02f42c8dadb4647bb2067dd2c5cc732" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 46127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:05.512741 788 mvcc.cc:205] Tried to move back new op lower bound from 7333997467656593408 to 7333997467527852032. Current Snapshot: MvccSnapshot[applied={T|T < 7333997467656593408}]
I20260927 16:32:05.514183 787 mvcc.cc:205] Tried to move back new op lower bound from 7333997467656593408 to 7333997467527852032. Current Snapshot: MvccSnapshot[applied={T|T < 7333997467656593408}]
I20260927 16:32:05.522065 32763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:05.522264 32763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:05.524672 32763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96babb2694de475295e519d4a3e62cce: Sending DeleteTablet for 3 replicas of tablet c6635c6d9e1b4335a3ee000c539f16ea
I20260927 16:32:05.525400 690 tablet_service.cc:1514] Processing DeleteTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:05 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:45908
I20260927 16:32:05.525401 422 tablet_service.cc:1514] Processing DeleteTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:05 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35778
I20260927 16:32:05.525529 556 tablet_service.cc:1514] Processing DeleteTablet for tablet c6635c6d9e1b4335a3ee000c539f16ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:05 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:53360
I20260927 16:32:05.525856 797 tablet_replica.cc:330] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36: stopping tablet replica
I20260927 16:32:05.526000 796 tablet_replica.cc:330] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: stopping tablet replica
I20260927 16:32:05.526198 798 tablet_replica.cc:330] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: stopping tablet replica
I20260927 16:32:05.526206 797 raft_consensus.cc:2229] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:05.526513 797 raft_consensus.cc:2258] T c6635c6d9e1b4335a3ee000c539f16ea P f0269751dd7f410eac965f23506bce36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:05.526674 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 359
I20260927 16:32:05.526721 798 raft_consensus.cc:2229] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:32:05.527067 798 raft_consensus.cc:2258] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:05.527431 796 raft_consensus.cc:2229] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:05.527653 796 raft_consensus.cc:2258] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:05.529090 798 ts_tablet_manager.cc:1905] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:05.529090 796 ts_tablet_manager.cc:1905] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:05.531742 798 ts_tablet_manager.cc:1918] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:05.531742 796 ts_tablet_manager.cc:1918] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:05.531827 796 log.cc:1196] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/c6635c6d9e1b4335a3ee000c539f16ea
I20260927 16:32:05.531827 798 log.cc:1196] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/c6635c6d9e1b4335a3ee000c539f16ea
I20260927 16:32:05.532158 796 ts_tablet_manager.cc:1939] T c6635c6d9e1b4335a3ee000c539f16ea P e02f42c8dadb4647bb2067dd2c5cc732: Deleting consensus metadata
I20260927 16:32:05.532158 798 ts_tablet_manager.cc:1939] T c6635c6d9e1b4335a3ee000c539f16ea P 059c9dc3ec5947f380b0095ff1bf566d: Deleting consensus metadata
I20260927 16:32:05.533771 32752 catalog_manager.cc:4841] TS 059c9dc3ec5947f380b0095ff1bf566d (127.15.93.130:46007): tablet c6635c6d9e1b4335a3ee000c539f16ea (table test-table [id=bc5a839941e14ba785f8d4afd4da7681]) successfully deleted
I20260927 16:32:05.534229 32749 catalog_manager.cc:4841] TS e02f42c8dadb4647bb2067dd2c5cc732 (127.15.93.131:46127): tablet c6635c6d9e1b4335a3ee000c539f16ea (table test-table [id=bc5a839941e14ba785f8d4afd4da7681]) successfully deleted
W20260927 16:32:05.537166 32750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:32905 (error 108)
I20260927 16:32:05.537961 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 493
I20260927 16:32:05.538182 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.537877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:32905 (local address 127.0.0.1:35784)
0927 16:32:05.538020 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:05.538097 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 16:32:05.539327 32750 catalog_manager.cc:4584] TS f0269751dd7f410eac965f23506bce36 (127.15.93.129:32905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6635c6d9e1b4335a3ee000c539f16ea: Network error: Client connection negotiation failed: client connection to 127.15.93.129:32905: connect: Connection refused (error 111)
I20260927 16:32:05.543924 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 627
I20260927 16:32:05.549631 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 32733
2026-09-27T16:32:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2261 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T16:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:05Z Disabled control of system clock
I20260927 16:32:05.580785 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33595
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:05.687227 803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:05.687573 803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:05.687613 803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:05.691484 803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:05.691534 803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:05.691548 803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:05.691565 803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:05.691581 803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:05.696165 803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:33595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:05.697129 803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:05.698410 803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:05.706518 808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:05.706530 809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.706584 803 server_base.cc:1034] running on GCE node
W20260927 16:32:05.706530 811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.707290 803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:05.708330 803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:05.709486 803 hybrid_clock.cc:648] HybridClock initialized: now 1790526725709465 us; error 36 us; skew 500 ppm
I20260927 16:32:05.711175 803 webserver.cc:460] Webserver started at http://127.15.93.190:39545/ using document root <none> and password file <none>
I20260927 16:32:05.711746 803 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:05.711808 803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:05.711963 803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:05.713979 803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "2d306a09ab264d268d868302d4889fec"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "9cb9c820750bf2fea09322e7857fb129"
server_key_iv: "e39dbb0d8cebfd699b4053230db80100"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.714483 803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "2d306a09ab264d268d868302d4889fec"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "9cb9c820750bf2fea09322e7857fb129"
server_key_iv: "e39dbb0d8cebfd699b4053230db80100"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.717905 803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:32:05.720127 817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.721272 803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:05.721364 803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "2d306a09ab264d268d868302d4889fec"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "9cb9c820750bf2fea09322e7857fb129"
server_key_iv: "e39dbb0d8cebfd699b4053230db80100"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.721441 803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:05.732002 803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:05.732559 803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:05.732703 803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:05.739583 803 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:33595
I20260927 16:32:05.739621 869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:33595 every 8 connection(s)
I20260927 16:32:05.740517 803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:05.743727 870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:05.745893 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 803
I20260927 16:32:05.746025 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:05.746338 15734 external_mini_cluster.cc:1428] Setting key b693e20a5f21d8d48ab908cdaf559b03
I20260927 16:32:05.750030 870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: Bootstrap starting.
I20260927 16:32:05.752249 870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:05.753015 870 log.cc:824] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:05.755034 870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: No bootstrap required, opened a new log
I20260927 16:32:05.755882 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:05.747787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.15.93.190:33595)
0927 16:32:05.748418 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:05.748435 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:05.748470 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:32:05.749211 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:05.749232 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:05.749658 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:05.750013 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:05.750466 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.751459 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.752340 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:05.752842 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:05.753021 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:05.753887 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:05.753893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:05.753916 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:05.754322 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:05.754380 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:05.754400 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:05.754541 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:05.754701 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:32:05.754965 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":150,"threads_started":1}
I20260927 16:32:05.758765 870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } }
I20260927 16:32:05.759023 870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:05.759102 870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d306a09ab264d268d868302d4889fec, State: Initialized, Role: FOLLOWER
I20260927 16:32:05.759603 870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } }
I20260927 16:32:05.759753 870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:05.759838 870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:05.759936 870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:05.761036 870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } }
I20260927 16:32:05.761377 870 leader_election.cc:302] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d306a09ab264d268d868302d4889fec; no voters:
I20260927 16:32:05.761683 870 leader_election.cc:288] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:05.761806 875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:05.762023 875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [term 1 LEADER]: Becoming Leader. State: Replica: 2d306a09ab264d268d868302d4889fec, State: Running, Role: LEADER
I20260927 16:32:05.762336 875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } }
I20260927 16:32:05.762877 870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:05.763978 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d306a09ab264d268d868302d4889fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } } }
I20260927 16:32:05.764107 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:05.764390 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d306a09ab264d268d868302d4889fec. Latest consensus state: current_term: 1 leader_uuid: "2d306a09ab264d268d868302d4889fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d306a09ab264d268d868302d4889fec" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 33595 } } }
I20260927 16:32:05.764488 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:05.764613 884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:05.767777 884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:05.773447 884 catalog_manager.cc:1347] Generated new cluster ID: d38ea8363e47450d931c3b23e13e064f
I20260927 16:32:05.773524 884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:05.780941 884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:05.781783 884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:05.788918 884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: Generated new TSK 0
I20260927 16:32:05.789595 884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:05.799602 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--builtin_ntp_servers=127.15.93.148:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:05.928665 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:05.928903 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:05.928941 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:05.932930 894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:05.932986 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:05.933074 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:05.937847 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:05.938867 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:05.940294 894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:05.949648 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:05.949648 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.949800 894 server_base.cc:1034] running on GCE node
W20260927 16:32:05.949657 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:05.950453 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:05.951036 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:05.952322 894 hybrid_clock.cc:648] HybridClock initialized: now 1790526725952276 us; error 57 us; skew 500 ppm
I20260927 16:32:05.954206 894 webserver.cc:460] Webserver started at http://127.15.93.129:35789/ using document root <none> and password file <none>
I20260927 16:32:05.954807 894 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:05.954901 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:05.955116 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:05.957285 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "2169981dce284081ac64ffe4e92cc7e0"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "6cc3d16bc6f7108d421b08ef77bc9969"
server_key_iv: "72d4b89b8aa896063a9d0f5762dd9fc0"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.957886 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "2169981dce284081ac64ffe4e92cc7e0"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "6cc3d16bc6f7108d421b08ef77bc9969"
server_key_iv: "72d4b89b8aa896063a9d0f5762dd9fc0"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.961575 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:05.964389 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.965653 894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 16:32:05.965780 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "2169981dce284081ac64ffe4e92cc7e0"
format_stamp: "Formatted at 2026-09-27 16:32:05 on dist-test-slave-grzf"
server_key: "6cc3d16bc6f7108d421b08ef77bc9969"
server_key_iv: "72d4b89b8aa896063a9d0f5762dd9fc0"
server_key_version: "encryptionkey@0"
I20260927 16:32:05.965909 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:05.992710 894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:05.993440 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:05.993651 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:05.994251 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:05.995200 894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:05.995272 894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:05.995347 894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:05.995405 894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.004042 894 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:36679
I20260927 16:32:06.004074 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:36679 every 8 connection(s)
I20260927 16:32:06.004951 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:06.006650 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 894
I20260927 16:32:06.006759 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:06.007085 15734 external_mini_cluster.cc:1428] Setting key 46e9fb41ecdd3aa7683122c55d96b343
I20260927 16:32:06.009183 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--builtin_ntp_servers=127.15.93.148:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:06.015810 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.006884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:58613 (local address 127.15.93.190:33595)
0927 16:32:06.007023 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.007035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.008401 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 16:32:06.009981 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.009988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.010048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.010141 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:06.011008 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.011846 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.013655 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.013867 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.013949 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.015301 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:06.015304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:06.015307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:06.015319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:06.015337 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:06.015344 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:06.015468 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.015643 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.015695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:06.017135 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.007441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33595 (local address 127.15.93.129:58613)
0927 16:32:06.008232 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.008287 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.009762 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.010295 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.010302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.010387 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:06.010870 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.010881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.011999 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.012006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.013524 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.013536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.014042 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.014048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.014284 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.014970 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:06.014975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:06.015037 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:06.015131 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:06.015470 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:06.015483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:06.015491 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.015642 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.015989 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260927 16:32:06.018474 1022 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33595
I20260927 16:32:06.018728 1022 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:06.019260 1022 heartbeater.cc:511] Master 127.15.93.190:33595 requested a full tablet report, sending...
I20260927 16:32:06.020880 834 ts_manager.cc:194] Registered new tserver with Master: 2169981dce284081ac64ffe4e92cc7e0 (127.15.93.129:36679)
I20260927 16:32:06.022013 834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.129:58613
W20260927 16:32:06.121505 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:06.121788 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:06.121863 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:06.126005 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:06.126078 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:06.126163 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:06.130824 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:06.131982 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:06.133333 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:06.143847 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:06.143852 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:06.144287 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:06.144665 1025 server_base.cc:1034] running on GCE node
I20260927 16:32:06.145104 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:06.146055 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:06.147218 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790526726147185 us; error 48 us; skew 500 ppm
I20260927 16:32:06.149086 1025 webserver.cc:460] Webserver started at http://127.15.93.130:35101/ using document root <none> and password file <none>
I20260927 16:32:06.149682 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:06.149742 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:06.149977 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:06.152122 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "d6c09d680db444c181a3b2781d0be7f7"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "4f0b95be119cd4857049694e24290f07"
server_key_iv: "8b62cce5fc35e1d4ec1be7ac7f8316dd"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.152667 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "d6c09d680db444c181a3b2781d0be7f7"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "4f0b95be119cd4857049694e24290f07"
server_key_iv: "8b62cce5fc35e1d4ec1be7ac7f8316dd"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.156453 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 16:32:06.158993 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.160359 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:32:06.160494 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "d6c09d680db444c181a3b2781d0be7f7"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "4f0b95be119cd4857049694e24290f07"
server_key_iv: "8b62cce5fc35e1d4ec1be7ac7f8316dd"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.160629 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:06.172469 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:06.173162 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:06.173357 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:06.173942 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:06.174894 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:06.174963 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.175035 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:06.175086 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.183312 1025 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:35865
I20260927 16:32:06.183421 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:35865 every 8 connection(s)
I20260927 16:32:06.184237 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:06.186398 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1025
I20260927 16:32:06.186514 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:06.186866 15734 external_mini_cluster.cc:1428] Setting key 6521bf943bb6feaf5a6343640e03252d
I20260927 16:32:06.188890 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--builtin_ntp_servers=127.15.93.148:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:06.194453 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.186060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:34213 (local address 127.15.93.190:33595)
0927 16:32:06.186204 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.186208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.187586 (+ 1378us) server_negotiation.cc:407] Connection header received
0927 16:32:06.189272 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.189275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.189325 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.189408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:06.190325 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.190855 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.192082 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.192466 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.192600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.193829 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:06.193835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:06.193839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:06.193857 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:06.193886 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:06.193899 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:06.193989 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.194142 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.194216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 16:32:06.195220 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.186587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33595 (local address 127.15.93.130:34213)
0927 16:32:06.187409 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.187471 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.189051 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.189602 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.189609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.189697 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:06.190190 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.190201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.190981 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.190985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.191951 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.191961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.192628 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.192632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.192787 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.193477 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:06.193479 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:06.193525 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:06.193645 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:06.193993 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:06.194006 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:06.194010 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.194106 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.194332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":144,"threads_started":1}
I20260927 16:32:06.196439 1153 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33595
I20260927 16:32:06.196717 1153 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:06.197407 1153 heartbeater.cc:511] Master 127.15.93.190:33595 requested a full tablet report, sending...
I20260927 16:32:06.198609 834 ts_manager.cc:194] Registered new tserver with Master: d6c09d680db444c181a3b2781d0be7f7 (127.15.93.130:35865)
I20260927 16:32:06.199199 834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.130:34213
W20260927 16:32:06.304204 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:06.304467 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:06.304529 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:06.308610 1156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:06.308669 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:06.308723 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:06.313578 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:06.314775 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:06.316220 1156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:06.325338 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:06.325357 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:06.325344 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:06.325515 1156 server_base.cc:1034] running on GCE node
I20260927 16:32:06.326056 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:06.326687 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:06.327868 1156 hybrid_clock.cc:648] HybridClock initialized: now 1790526726327852 us; error 27 us; skew 500 ppm
I20260927 16:32:06.329594 1156 webserver.cc:460] Webserver started at http://127.15.93.131:40561/ using document root <none> and password file <none>
I20260927 16:32:06.330159 1156 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:06.330255 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:06.330482 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:06.332537 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "e7394c8d11c84cd1b42435af4c10fecf"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "16a28de7addbb42a7108056b9c0a2bbe"
server_key_iv: "b18ebece2708415b34cfbe0966a43fe3"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.333097 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "e7394c8d11c84cd1b42435af4c10fecf"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "16a28de7addbb42a7108056b9c0a2bbe"
server_key_iv: "b18ebece2708415b34cfbe0966a43fe3"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.336787 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:06.339299 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.340555 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:06.340690 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "e7394c8d11c84cd1b42435af4c10fecf"
format_stamp: "Formatted at 2026-09-27 16:32:06 on dist-test-slave-grzf"
server_key: "16a28de7addbb42a7108056b9c0a2bbe"
server_key_iv: "b18ebece2708415b34cfbe0966a43fe3"
server_key_version: "encryptionkey@0"
I20260927 16:32:06.340807 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:06.351019 1156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:06.351747 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:06.351933 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:06.352492 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:06.353410 1156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:06.353480 1156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.353544 1156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:06.353596 1156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:06.362046 1156 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40877
I20260927 16:32:06.362066 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40877 every 8 connection(s)
I20260927 16:32:06.362916 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:32:06.366601 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1156
I20260927 16:32:06.366725 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:06.367017 15734 external_mini_cluster.cc:1428] Setting key 3c88a7cd87f19e005b222f41b6200194
I20260927 16:32:06.372901 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.365021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:36847 (local address 127.15.93.190:33595)
0927 16:32:06.365163 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.365167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.366245 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 16:32:06.368086 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.368090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.368146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.368244 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:06.369356 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.369883 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.371085 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.371298 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.371436 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.372509 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:06.372512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:06.372514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:06.372525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:06.372543 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:06.372552 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:06.372608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.372746 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.372794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:32:06.373769 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.365460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:33595 (local address 127.15.93.131:36847)
0927 16:32:06.366097 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.366137 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.367749 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.368474 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.368492 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.368584 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:06.369183 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.369201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.370024 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.370027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.370980 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.370990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.371473 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.371477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.371619 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.372259 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:06.372261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:06.372303 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:06.372365 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:06.372626 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:06.372633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:06.372637 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.372733 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.372965 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260927 16:32:06.374876 1284 heartbeater.cc:347] Connected to a master server at 127.15.93.190:33595
I20260927 16:32:06.375077 1284 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:06.375605 1284 heartbeater.cc:511] Master 127.15.93.190:33595 requested a full tablet report, sending...
I20260927 16:32:06.376554 834 ts_manager.cc:194] Registered new tserver with Master: e7394c8d11c84cd1b42435af4c10fecf (127.15.93.131:40877)
I20260927 16:32:06.377068 834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.131:36847
I20260927 16:32:06.380896 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:06.386390 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.382806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.15.93.190:33595)
0927 16:32:06.382930 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.382934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.383136 (+ 202us) server_negotiation.cc:407] Connection header received
0927 16:32:06.383233 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.383235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.383270 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.383335 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:06.383597 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.384089 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.384843 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.385054 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.385128 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.385966 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:06.385969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:06.385971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:06.385982 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:06.386002 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:06.386009 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:06.386078 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.386237 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.386286 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:06.389125 834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:06.392213 834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:06.405879 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.400906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.15.93.131:40877)
0927 16:32:06.401614 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.401635 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.401764 (+ 129us) server_negotiation.cc:407] Connection header received
0927 16:32:06.401884 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.401887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.402031 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.402182 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:06.402463 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403218 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.404678 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.405214 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.405336 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.405533 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.405625 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.405715 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":182,"threads_started":1}
I20260927 16:32:06.405885 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.400506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40877 (local address 127.0.0.1:34462)
0927 16:32:06.401645 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.401657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.401768 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.402182 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.402188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.402209 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:06.402340 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.402347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403330 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.403336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.404556 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.404566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.405336 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.405340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.405440 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.405475 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.405590 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.405710 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":99,"threads_started":1}
I20260927 16:32:06.406687 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.400239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:36679 (local address 127.0.0.1:54110)
0927 16:32:06.401223 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.401283 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.401407 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.402196 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.402198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.402224 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:06.402362 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.402368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403871 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.403874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.405241 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.405252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.406314 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.406318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.406403 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.406425 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.406487 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.406549 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":73,"spinlock_wait_cycles":9344,"thread_start_us":86,"threads_started":1}
I20260927 16:32:06.406838 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.401080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.15.93.129:36679)
0927 16:32:06.401704 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.401715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.401738 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:32:06.401814 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.401820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.402024 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.402507 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:06.402647 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403754 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.405381 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.406177 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.406310 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.406494 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.406545 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.406654 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":115,"threads_started":1}
I20260927 16:32:06.407837 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.400239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35865 (local address 127.0.0.1:42974)
0927 16:32:06.401356 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.401370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.401468 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.402328 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.402330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.402345 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:06.402414 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.402420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403870 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.403874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.405241 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.405252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.406663 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.406667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.407609 (+ 942us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.407646 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.407692 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.407742 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":188,"spinlock_wait_cycles":4480,"thread_start_us":90,"threads_started":1}
I20260927 16:32:06.408476 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.400384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.15.93.130:35865)
0927 16:32:06.401820 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.401830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.401848 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:32:06.401911 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.401915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.402051 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.402189 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:06.402529 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.403737 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.405381 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.406523 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.408124 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.408257 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.408295 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.408360 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":98,"threads_started":1}
I20260927 16:32:06.409171 1218 tablet_service.cc:1467] Processing CreateTablet for tablet 648193b3871248c285bef20e12d22a1a (DEFAULT_TABLE table=test-table [id=d1b26a22fe1248f29ddb8e0c1f8a29f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:06.409483 956 tablet_service.cc:1467] Processing CreateTablet for tablet 648193b3871248c285bef20e12d22a1a (DEFAULT_TABLE table=test-table [id=d1b26a22fe1248f29ddb8e0c1f8a29f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:06.410336 1218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648193b3871248c285bef20e12d22a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:06.410370 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648193b3871248c285bef20e12d22a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:06.410733 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 648193b3871248c285bef20e12d22a1a (DEFAULT_TABLE table=test-table [id=d1b26a22fe1248f29ddb8e0c1f8a29f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:06.411702 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648193b3871248c285bef20e12d22a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:06.417462 1301 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Bootstrap starting.
I20260927 16:32:06.418565 1302 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: Bootstrap starting.
I20260927 16:32:06.419622 1301 tablet_bootstrap.cc:654] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:06.419701 1303 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Bootstrap starting.
I20260927 16:32:06.420377 1301 log.cc:824] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:06.420517 1302 tablet_bootstrap.cc:654] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:06.421319 1302 log.cc:824] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:06.421758 1303 tablet_bootstrap.cc:654] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:06.422569 1303 log.cc:824] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:06.422741 1301 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: No bootstrap required, opened a new log
I20260927 16:32:06.422928 1301 ts_tablet_manager.cc:1397] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:32:06.423228 1302 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: No bootstrap required, opened a new log
I20260927 16:32:06.423454 1302 ts_tablet_manager.cc:1397] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:32:06.424392 1303 tablet_bootstrap.cc:492] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: No bootstrap required, opened a new log
I20260927 16:32:06.424638 1303 ts_tablet_manager.cc:1397] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:32:06.427456 1302 raft_consensus.cc:348] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.427419 1301 raft_consensus.cc:348] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.427695 1302 raft_consensus.cc:374] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:06.427696 1301 raft_consensus.cc:374] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:06.427753 1302 raft_consensus.cc:729] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2169981dce284081ac64ffe4e92cc7e0, State: Initialized, Role: FOLLOWER
I20260927 16:32:06.427760 1301 raft_consensus.cc:729] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c09d680db444c181a3b2781d0be7f7, State: Initialized, Role: FOLLOWER
I20260927 16:32:06.427819 1303 raft_consensus.cc:348] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.428088 1303 raft_consensus.cc:374] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:06.428145 1303 raft_consensus.cc:729] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7394c8d11c84cd1b42435af4c10fecf, State: Initialized, Role: FOLLOWER
I20260927 16:32:06.428205 1301 consensus_queue.cc:260] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.428205 1302 consensus_queue.cc:260] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.428856 1153 heartbeater.cc:503] Master 127.15.93.190:33595 was elected leader, sending a full tablet report...
I20260927 16:32:06.429109 1022 heartbeater.cc:503] Master 127.15.93.190:33595 was elected leader, sending a full tablet report...
I20260927 16:32:06.429184 1302 ts_tablet_manager.cc:1428] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:32:06.428618 1303 consensus_queue.cc:260] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.430522 1303 ts_tablet_manager.cc:1428] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:06.430887 1284 heartbeater.cc:503] Master 127.15.93.190:33595 was elected leader, sending a full tablet report...
I20260927 16:32:06.431358 1307 raft_consensus.cc:482] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:06.431519 1301 ts_tablet_manager.cc:1428] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260927 16:32:06.431521 1307 raft_consensus.cc:504] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.433983 1307 leader_election.cc:288] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c09d680db444c181a3b2781d0be7f7 (127.15.93.130:35865), e7394c8d11c84cd1b42435af4c10fecf (127.15.93.131:40877)
W20260927 16:32:06.435140 1154 tablet.cc:2369] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:06.438380 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.434202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40877 (local address 127.15.93.129:42663)
0927 16:32:06.434559 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.434572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.434669 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.434942 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.434945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.434965 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:06.435101 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.435108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.436105 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.436108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.437151 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.437161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.437995 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.437997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.438072 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.438101 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.438164 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.438214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":94,"threads_started":1}
I20260927 16:32:06.438401 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.434269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:42663 (local address 127.15.93.131:40877)
0927 16:32:06.434385 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.434389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.434655 (+ 266us) server_negotiation.cc:407] Connection header received
0927 16:32:06.434772 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.434775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.434818 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.434891 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:06.435221 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.435985 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.437270 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.437878 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.437972 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.438003 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.438210 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.438288 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:32:06.438474 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.433658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35865 (local address 127.15.93.129:56597)
0927 16:32:06.433760 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.433775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:06.433856 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:06.434260 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:06.434265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:06.434289 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:06.434409 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.434437 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.435404 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.435408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.436424 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:06.436433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.438065 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.438068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:06.438140 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.438191 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 16:32:06.438267 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 16:32:06.438329 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 16:32:06.438481 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.433816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:56597 (local address 127.15.93.130:35865)
0927 16:32:06.433947 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.433953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.433966 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:32:06.434007 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.434010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.434060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.434147 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:06.434545 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.435275 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.436537 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.437155 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.437292 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.437344 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.438299 (+ 955us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.438368 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:32:06.439046 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648193b3871248c285bef20e12d22a1a" candidate_uuid: "2169981dce284081ac64ffe4e92cc7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c09d680db444c181a3b2781d0be7f7" is_pre_election: true
I20260927 16:32:06.439158 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648193b3871248c285bef20e12d22a1a" candidate_uuid: "2169981dce284081ac64ffe4e92cc7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7394c8d11c84cd1b42435af4c10fecf" is_pre_election: true
I20260927 16:32:06.439328 1107 raft_consensus.cc:2454] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2169981dce284081ac64ffe4e92cc7e0 in term 0.
I20260927 16:32:06.439473 1238 raft_consensus.cc:2454] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2169981dce284081ac64ffe4e92cc7e0 in term 0.
I20260927 16:32:06.439831 911 leader_election.cc:302] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2169981dce284081ac64ffe4e92cc7e0, d6c09d680db444c181a3b2781d0be7f7; no voters:
I20260927 16:32:06.440095 1307 raft_consensus.cc:2781] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:06.440177 1307 raft_consensus.cc:482] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:06.440219 1307 raft_consensus.cc:3037] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:06.441172 1307 raft_consensus.cc:504] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.441491 1307 leader_election.cc:288] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [CANDIDATE]: Term 1 election: Requested vote from peers d6c09d680db444c181a3b2781d0be7f7 (127.15.93.130:35865), e7394c8d11c84cd1b42435af4c10fecf (127.15.93.131:40877)
I20260927 16:32:06.441809 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648193b3871248c285bef20e12d22a1a" candidate_uuid: "2169981dce284081ac64ffe4e92cc7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c09d680db444c181a3b2781d0be7f7"
I20260927 16:32:06.441857 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648193b3871248c285bef20e12d22a1a" candidate_uuid: "2169981dce284081ac64ffe4e92cc7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7394c8d11c84cd1b42435af4c10fecf"
I20260927 16:32:06.441917 1107 raft_consensus.cc:3037] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:06.441989 1238 raft_consensus.cc:3037] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:06.442901 1107 raft_consensus.cc:2454] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2169981dce284081ac64ffe4e92cc7e0 in term 1.
I20260927 16:32:06.442981 1238 raft_consensus.cc:2454] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2169981dce284081ac64ffe4e92cc7e0 in term 1.
I20260927 16:32:06.443225 911 leader_election.cc:302] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2169981dce284081ac64ffe4e92cc7e0, d6c09d680db444c181a3b2781d0be7f7; no voters:
I20260927 16:32:06.443404 1307 raft_consensus.cc:2781] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:06.443635 1307 raft_consensus.cc:686] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2169981dce284081ac64ffe4e92cc7e0, State: Running, Role: LEADER
I20260927 16:32:06.443931 1307 consensus_queue.cc:237] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } }
I20260927 16:32:06.446859 833 catalog_manager.cc:5495] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2169981dce284081ac64ffe4e92cc7e0 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "2169981dce284081ac64ffe4e92cc7e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2169981dce284081ac64ffe4e92cc7e0" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 } health_report { overall_health: UNKNOWN } } }
I20260927 16:32:06.468407 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.465179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.15.93.129:36679)
0927 16:32:06.465306 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:06.465310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:06.465324 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:32:06.465446 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:06.465449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:06.465505 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:06.465608 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:06.465766 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.466246 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.467027 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:06.467268 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:06.467343 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:06.467664 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:06.468049 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:06.468141 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:06.468196 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:32:06.468238 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:32:06.474279 1238 raft_consensus.cc:1261] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 1 FOLLOWER]: Refusing update from remote peer 2169981dce284081ac64ffe4e92cc7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:06.474279 1107 raft_consensus.cc:1261] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 1 FOLLOWER]: Refusing update from remote peer 2169981dce284081ac64ffe4e92cc7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:06.475072 1307 consensus_queue.cc:1035] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c09d680db444c181a3b2781d0be7f7" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:06.475314 1311 consensus_queue.cc:1035] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7394c8d11c84cd1b42435af4c10fecf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:06.490918 833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:06.491091 833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:06.493351 833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d306a09ab264d268d868302d4889fec: Sending DeleteTablet for 3 replicas of tablet 648193b3871248c285bef20e12d22a1a
I20260927 16:32:06.494096 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 648193b3871248c285bef20e12d22a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:06 UTC) from {username='slave'} at 127.0.0.1:54110
I20260927 16:32:06.494127 1087 tablet_service.cc:1514] Processing DeleteTablet for tablet 648193b3871248c285bef20e12d22a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:06 UTC) from {username='slave'} at 127.0.0.1:42974
I20260927 16:32:06.494153 1218 tablet_service.cc:1514] Processing DeleteTablet for tablet 648193b3871248c285bef20e12d22a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:06 UTC) from {username='slave'} at 127.0.0.1:34462
I20260927 16:32:06.494593 1324 tablet_replica.cc:330] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: stopping tablet replica
I20260927 16:32:06.494796 1323 tablet_replica.cc:330] T 648193b3871248c285bef20e12d22a1a P 2169981dce284081ac64ffe4e92cc7e0: stopping tablet replica
I20260927 16:32:06.494612 1325 tablet_replica.cc:330] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: stopping tablet replica
I20260927 16:32:06.494921 1324 raft_consensus.cc:2229] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:06.495191 1325 raft_consensus.cc:2229] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:06.495229 1324 raft_consensus.cc:2258] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:06.495548 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 894
I20260927 16:32:06.497198 1325 raft_consensus.cc:2258] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:06.498637 1325 ts_tablet_manager.cc:1905] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:06.500445 1324 ts_tablet_manager.cc:1905] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:06.502118 1325 ts_tablet_manager.cc:1918] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:06.502224 1325 log.cc:1196] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/648193b3871248c285bef20e12d22a1a
I20260927 16:32:06.502537 1325 ts_tablet_manager.cc:1939] T 648193b3871248c285bef20e12d22a1a P e7394c8d11c84cd1b42435af4c10fecf: Deleting consensus metadata
I20260927 16:32:06.502740 1324 ts_tablet_manager.cc:1918] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:06.502848 1324 log.cc:1196] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/648193b3871248c285bef20e12d22a1a
I20260927 16:32:06.503165 1324 ts_tablet_manager.cc:1939] T 648193b3871248c285bef20e12d22a1a P d6c09d680db444c181a3b2781d0be7f7: Deleting consensus metadata
W20260927 16:32:06.503501 821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:36679 (error 108)
I20260927 16:32:06.504031 818 catalog_manager.cc:4841] TS e7394c8d11c84cd1b42435af4c10fecf (127.15.93.131:40877): tablet 648193b3871248c285bef20e12d22a1a (table test-table [id=d1b26a22fe1248f29ddb8e0c1f8a29f3]) successfully deleted
I20260927 16:32:06.504457 820 catalog_manager.cc:4841] TS d6c09d680db444c181a3b2781d0be7f7 (127.15.93.130:35865): tablet 648193b3871248c285bef20e12d22a1a (table test-table [id=d1b26a22fe1248f29ddb8e0c1f8a29f3]) successfully deleted
I20260927 16:32:06.504504 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:06.504213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:36679 (local address 127.0.0.1:54138)
0927 16:32:06.504323 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:06.504417 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:36679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 16:32:06.504659 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1025
W20260927 16:32:06.504819 821 catalog_manager.cc:4584] TS 2169981dce284081ac64ffe4e92cc7e0 (127.15.93.129:36679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 648193b3871248c285bef20e12d22a1a: Network error: Client connection negotiation failed: client connection to 127.15.93.129:36679: connect: Connection refused (error 111)
I20260927 16:32:06.510278 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1156
I20260927 16:32:06.515692 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 803
2026-09-27T16:32:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:06 dist-test-slave-grzf krb5kdc[1329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:06 dist-test-slave-grzf krb5kdc[1329](info): set up 2 sockets
Sep 27 16:32:06 dist-test-slave-grzf krb5kdc[1329](info): commencing operation
krb5kdc: starting...
W20260927 16:32:07.563284 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:07 dist-test-slave-grzf krb5kdc[1329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:07.680732 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38687
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:07.788506 1342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:07.788769 1342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:07.788810 1342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:07.792841 1342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:07.792894 1342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:07.792914 1342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:07.792932 1342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:07.792982 1342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:07.798338 1342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:38687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:07.799409 1342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:07.800459 1342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:07.808698 1348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:07.808707 1347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:07.808707 1350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:07.809052 1342 server_base.cc:1034] running on GCE node
I20260927 16:32:07.809638 1342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:07.810695 1342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:07.811854 1342 hybrid_clock.cc:648] HybridClock initialized: now 1790526727811826 us; error 43 us; skew 500 ppm
Sep 27 16:32:07 dist-test-slave-grzf krb5kdc[1329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526727, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:07.814702 1342 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:07.815608 1342 webserver.cc:460] Webserver started at http://127.15.93.190:32935/ using document root <none> and password file <none>
I20260927 16:32:07.816187 1342 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:07.816246 1342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:07.816428 1342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:07.818500 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6"
format_stamp: "Formatted at 2026-09-27 16:32:07 on dist-test-slave-grzf"
server_key: "a88c9432a660d330ecc8d4e7a8fc9bd3"
server_key_iv: "c8b4b31cb5808bfe88c5bc1f5990e397"
server_key_version: "encryptionkey@0"
I20260927 16:32:07.818991 1342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6"
format_stamp: "Formatted at 2026-09-27 16:32:07 on dist-test-slave-grzf"
server_key: "a88c9432a660d330ecc8d4e7a8fc9bd3"
server_key_iv: "c8b4b31cb5808bfe88c5bc1f5990e397"
server_key_version: "encryptionkey@0"
I20260927 16:32:07.822556 1342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:07.824919 1357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:07.826171 1342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:07.826287 1342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6"
format_stamp: "Formatted at 2026-09-27 16:32:07 on dist-test-slave-grzf"
server_key: "a88c9432a660d330ecc8d4e7a8fc9bd3"
server_key_iv: "c8b4b31cb5808bfe88c5bc1f5990e397"
server_key_version: "encryptionkey@0"
I20260927 16:32:07.826385 1342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:07.846098 1342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:07.848974 1342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:07.849187 1342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:07.856210 1342 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:38687
I20260927 16:32:07.856237 1409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:38687 every 8 connection(s)
I20260927 16:32:07.857120 1342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:07.859795 1410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:07.865154 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: Bootstrap starting.
I20260927 16:32:07.866811 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1342
I20260927 16:32:07.866959 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:07.867256 15734 external_mini_cluster.cc:1428] Setting key 82a6be188c4af91ac6e2fecd82d6b1f9
I20260927 16:32:07.867862 1410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:07.868765 1410 log.cc:824] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:07.870810 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: No bootstrap required, opened a new log
I20260927 16:32:07.874368 1410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } }
I20260927 16:32:07.874684 1410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:07.874770 1410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0aac07e9a04dc4a6a93b84268e2ec6, State: Initialized, Role: FOLLOWER
I20260927 16:32:07.875236 1410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } }
I20260927 16:32:07.875401 1410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:07.875470 1410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:07.875567 1410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 16:32:07 dist-test-slave-grzf krb5kdc[1329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:07.876523 1410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } }
I20260927 16:32:07.876879 1410 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0aac07e9a04dc4a6a93b84268e2ec6; no voters:
I20260927 16:32:07.877276 1410 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:07.877341 1415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:07.877810 1415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0aac07e9a04dc4a6a93b84268e2ec6, State: Running, Role: LEADER
I20260927 16:32:07.878116 1415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } }
I20260927 16:32:07.878575 1410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:07.879766 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } } }
I20260927 16:32:07.879894 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:07.880100 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0aac07e9a04dc4a6a93b84268e2ec6. Latest consensus state: current_term: 1 leader_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0aac07e9a04dc4a6a93b84268e2ec6" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 38687 } } }
I20260927 16:32:07.880182 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:07.882144 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:07.868425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.15.93.190:38687)
0927 16:32:07.869024 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:07.869039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:07.869074 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:32:07.869895 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:07.869915 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:07.870246 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:07.870586 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:07.871424 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:07.872367 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:07.873236 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:07.873708 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:07.873889 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:07.876563 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:07.876568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:07.876583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:07.876608 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:07.878571 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:07.879011 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:07.879019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:07.879024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:07.879160 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:07.879515 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:07.879518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:07.879520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:07.880011 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:07.880897 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:07.881173 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 16:32:07.881444 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":180,"threads_started":1}
W20260927 16:32:07.883642 1430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:07.883719 1430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:07.883776 1424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:07.884552 1424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:07.889915 1424 catalog_manager.cc:1347] Generated new cluster ID: 373fd57efd514ef8930c201bb763217b
I20260927 16:32:07.889983 1424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:07.908747 1424 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:07.910009 1424 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:07.923489 1424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: Generated new TSK 0
I20260927 16:32:07.924304 1424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:32:07.952695 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:08.062889 1436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:08.063151 1436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:08.063194 1436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:08.067407 1436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:08.067485 1436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:08.067548 1436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:08.073019 1436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:08.074033 1436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:08.075018 1436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:08.084848 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:08.084867 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:08.084916 1436 server_base.cc:1034] running on GCE node
W20260927 16:32:08.084832 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:08.085508 1436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:08.086162 1436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:08.087435 1436 hybrid_clock.cc:648] HybridClock initialized: now 1790526728087346 us; error 48 us; skew 500 ppm
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:08.090201 1436 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:32:08.091032 1436 webserver.cc:460] Webserver started at http://127.15.93.129:36369/ using document root <none> and password file <none>
I20260927 16:32:08.091650 1436 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:08.091739 1436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:08.091965 1436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:08.093910 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "180f4270d5104f069beff891795c860d"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "da68e23cb9a114c3bf91aa5b863a2e32"
server_key_iv: "0075ae6b925c7014cf3baf5a64ecb5d7"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.094475 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "180f4270d5104f069beff891795c860d"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "da68e23cb9a114c3bf91aa5b863a2e32"
server_key_iv: "0075ae6b925c7014cf3baf5a64ecb5d7"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.097982 1436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:32:08.100270 1451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.101486 1436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:08.101604 1436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "180f4270d5104f069beff891795c860d"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "da68e23cb9a114c3bf91aa5b863a2e32"
server_key_iv: "0075ae6b925c7014cf3baf5a64ecb5d7"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.101698 1436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:08.111968 1436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:08.114741 1436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:08.114938 1436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:08.115557 1436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:08.116357 1436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:08.116402 1436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.116472 1436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:08.116504 1436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.125154 1436 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:40607
I20260927 16:32:08.125231 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:40607 every 8 connection(s)
I20260927 16:32:08.126072 1436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:08.128877 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1436
I20260927 16:32:08.129014 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:08.129365 15734 external_mini_cluster.cc:1428] Setting key f042c816938b3ee995bb8071ac100418
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:32:08.141098 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.127857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:45209 (local address 127.15.93.190:38687)
0927 16:32:08.128023 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.128026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.128819 (+ 793us) server_negotiation.cc:407] Connection header received
0927 16:32:08.130592 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.130596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.130649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.130725 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:08.132328 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.133073 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.134232 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.134676 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.134804 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.137331 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:08.137335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:08.137338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:08.137370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:08.138807 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.139321 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.139326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.139328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.139461 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.139807 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.139810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.139812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.139996 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:08.140130 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.140786 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.140912 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:32:08.141933 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.128147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38687 (local address 127.15.93.129:45209)
0927 16:32:08.128582 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.128689 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.130364 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.130924 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.130930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.131461 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:08.132167 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.132188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.133209 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.133212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.134100 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.134109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.134825 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.134828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.135044 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.135701 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:08.135717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:08.137167 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:08.138926 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.138931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.138946 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.139217 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.139559 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.139562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.139564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.139704 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.140130 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:08.140138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:08.140489 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.140730 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.141048 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
I20260927 16:32:08.143445 1565 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38687
I20260927 16:32:08.143715 1565 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:08.144330 1565 heartbeater.cc:511] Master 127.15.93.190:38687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:32:08.147102 1374 ts_manager.cc:194] Registered new tserver with Master: 180f4270d5104f069beff891795c860d (127.15.93.129:40607)
I20260927 16:32:08.148743 1374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:32:08.162580 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:08.271554 1570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:08.271816 1570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:08.271857 1570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:08.275728 1570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:08.275775 1570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:08.275830 1570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:08.281181 1570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:08.282116 1570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:08.283041 1570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:08.292258 1576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:08.292243 1578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:08.292246 1575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:08.292409 1570 server_base.cc:1034] running on GCE node
I20260927 16:32:08.292914 1570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:08.293557 1570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:08.294749 1570 hybrid_clock.cc:648] HybridClock initialized: now 1790526728294734 us; error 25 us; skew 500 ppm
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:08.297354 1570 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:32:08.298216 1570 webserver.cc:460] Webserver started at http://127.15.93.130:44293/ using document root <none> and password file <none>
I20260927 16:32:08.298756 1570 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:08.298844 1570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:08.299068 1570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:08.301064 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "3b2873b44f28417fb5f33885028b0389"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "e4d9da0289b51077392e9d9aec7fb080"
server_key_iv: "fd30f7c30054e3280997673bbb6919c9"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.301628 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "3b2873b44f28417fb5f33885028b0389"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "e4d9da0289b51077392e9d9aec7fb080"
server_key_iv: "fd30f7c30054e3280997673bbb6919c9"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.305342 1570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:08.307643 1585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.308840 1570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:32:08.308964 1570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "3b2873b44f28417fb5f33885028b0389"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "e4d9da0289b51077392e9d9aec7fb080"
server_key_iv: "fd30f7c30054e3280997673bbb6919c9"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.309072 1570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:08.322741 1570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:08.325455 1570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:08.325657 1570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:08.326211 1570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:08.326990 1570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:08.327059 1570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.327139 1570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:08.327186 1570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.335961 1570 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:46811
I20260927 16:32:08.335996 1698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:46811 every 8 connection(s)
I20260927 16:32:08.336853 1570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:08.338616 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1570
I20260927 16:32:08.338742 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:08.339053 15734 external_mini_cluster.cc:1428] Setting key cef3f028a39f3a5d1304b7b0c6559aaa
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:08.356187 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.338629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:38021 (local address 127.15.93.190:38687)
0927 16:32:08.338767 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.338774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.339843 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 16:32:08.341267 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.341271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.341325 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.341424 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:08.342571 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.343099 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.344413 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.344644 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.344754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.352334 (+ 7580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:08.352340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:08.352345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:08.352380 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:08.354044 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.354611 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.354613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.354615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.354691 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.355038 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.355041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.355043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.355237 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:08.355342 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.355889 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.356023 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:32:08.357089 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.339153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38687 (local address 127.15.93.130:38021)
0927 16:32:08.339705 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.339748 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.341059 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.341602 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.341609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.341955 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:08.342429 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.342442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.343245 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.343248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.344291 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.344301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.344793 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.344797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.344964 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.350288 (+ 5324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:08.350317 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:08.352125 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:08.354182 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.354187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.354202 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.354476 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.354787 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.354790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.354792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.354932 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.355353 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:08.355361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:08.355643 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.355834 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.356182 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":163,"threads_started":1}
I20260927 16:32:08.358294 1699 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38687
I20260927 16:32:08.358547 1699 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:08.359045 1699 heartbeater.cc:511] Master 127.15.93.190:38687 requested a full tablet report, sending...
I20260927 16:32:08.360174 1374 ts_manager.cc:194] Registered new tserver with Master: 3b2873b44f28417fb5f33885028b0389 (127.15.93.130:46811)
I20260927 16:32:08.360802 1374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:38021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:32:08.366324 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:08.475422 1704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:08.475672 1704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:08.475744 1704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:08.479812 1704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:08.479863 1704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:08.479979 1704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:08.485167 1704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:38687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:08.486129 1704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:08.487092 1704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:08.496034 1709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:08.496064 1712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:08.496068 1710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:08.496272 1704 server_base.cc:1034] running on GCE node
I20260927 16:32:08.496766 1704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:08.497301 1704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:08.498456 1704 hybrid_clock.cc:648] HybridClock initialized: now 1790526728498437 us; error 32 us; skew 500 ppm
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:08.501042 1704 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:32:08.501775 1704 webserver.cc:460] Webserver started at http://127.15.93.131:34455/ using document root <none> and password file <none>
I20260927 16:32:08.502246 1704 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:08.502300 1704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:08.502450 1704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:08.504388 1704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "e48c9d8dd2814db3a08341a7fb97ccbf"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "4d720875ca5cbcf4361c5678ee9ed80a"
server_key_iv: "dcea545a00762e71bfbea70b3ef32f61"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.504958 1704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "e48c9d8dd2814db3a08341a7fb97ccbf"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "4d720875ca5cbcf4361c5678ee9ed80a"
server_key_iv: "dcea545a00762e71bfbea70b3ef32f61"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.508428 1704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 16:32:08.510581 1719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.511814 1704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:08.511936 1704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "e48c9d8dd2814db3a08341a7fb97ccbf"
format_stamp: "Formatted at 2026-09-27 16:32:08 on dist-test-slave-grzf"
server_key: "4d720875ca5cbcf4361c5678ee9ed80a"
server_key_iv: "dcea545a00762e71bfbea70b3ef32f61"
server_key_version: "encryptionkey@0"
I20260927 16:32:08.512039 1704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:08.545890 1704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:08.548875 1704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:08.549082 1704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:08.549686 1704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:08.550504 1704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:08.550582 1704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.550653 1704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:08.550688 1704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:08.559041 1704 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:37815
I20260927 16:32:08.559074 1832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:37815 every 8 connection(s)
I20260927 16:32:08.559981 1704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:32:08.562288 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1704
I20260927 16:32:08.562431 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:08.562734 15734 external_mini_cluster.cc:1428] Setting key 6758225fe07696de1c367c52c4b4f220
Sep 27 16:32:08 dist-test-slave-grzf krb5kdc[1329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526728, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:08.573122 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.561668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:44787 (local address 127.15.93.190:38687)
0927 16:32:08.561938 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.561943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.562809 (+ 866us) server_negotiation.cc:407] Connection header received
0927 16:32:08.564293 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.564296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.564346 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.564415 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:08.565599 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.566119 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.567232 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.567462 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.567545 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.569855 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:08.569858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:08.569860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:08.569883 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:08.571373 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.571883 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.571886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.571887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.571961 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.572251 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.572253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.572254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.572414 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:08.572513 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.572912 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.573006 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260927 16:32:08.573905 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.562183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:38687 (local address 127.15.93.131:44787)
0927 16:32:08.562664 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.562715 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.564088 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.564616 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.564625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.565009 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:08.565456 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.565468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.566227 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.566229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.567130 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.567139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.567598 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.567601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.567736 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.568251 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:08.568265 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:08.569699 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:08.571547 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.571552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.571562 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.571801 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.572039 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:08.572041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:08.572042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:08.572176 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:08.572495 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:08.572500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:08.572717 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.572865 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.573109 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":144,"threads_started":1}
I20260927 16:32:08.575107 1833 heartbeater.cc:347] Connected to a master server at 127.15.93.190:38687
I20260927 16:32:08.575357 1833 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:08.576002 1833 heartbeater.cc:511] Master 127.15.93.190:38687 requested a full tablet report, sending...
I20260927 16:32:08.577103 1374 ts_manager.cc:194] Registered new tserver with Master: e48c9d8dd2814db3a08341a7fb97ccbf (127.15.93.131:37815)
I20260927 16:32:08.577654 1374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:44787
I20260927 16:32:08.588515 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:08.597266 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.590325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.15.93.190:38687)
0927 16:32:08.590492 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.590496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.590632 (+ 136us) server_negotiation.cc:407] Connection header received
0927 16:32:08.590698 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.590700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.590738 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.590800 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:08.591481 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.591989 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.592757 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.592959 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.593030 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.594443 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:08.594446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:08.594448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:08.594468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:08.595720 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.596173 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.596176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.596177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.596241 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:08.596494 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:08.596496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:08.596497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:08.596680 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:08.596788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.596987 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.597087 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 16:32:08.600006 1374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:08.602921 1374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:08.616719 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.611146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:37815 (local address 127.0.0.1:59792)
0927 16:32:08.611784 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.611800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.611891 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.612599 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.612602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.612618 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:08.612740 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.612747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.614177 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.614180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.615131 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.615143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.616201 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.616204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.616315 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.616354 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.616450 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.616566 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":67,"thread_start_us":86,"threads_started":1}
I20260927 16:32:08.616730 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.611887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.15.93.131:37815)
0927 16:32:08.612220 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.612230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.612247 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:32:08.612306 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.612310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.612446 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.612683 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:08.612853 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.614043 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.615324 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.616075 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.616201 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.616407 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.616471 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.616590 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":139,"threads_started":1}
I20260927 16:32:08.617460 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.610924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.15.93.130:46811)
0927 16:32:08.611611 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.611626 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.611651 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:32:08.611849 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.611856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.612130 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.612266 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:08.613080 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.613917 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.615718 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.616537 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.616617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.616822 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.617209 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.617331 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":157,"threads_started":1}
I20260927 16:32:08.617471 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.610874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:40607 (local address 127.0.0.1:41144)
0927 16:32:08.611533 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.611549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.611653 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.612695 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.612698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.612711 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:08.612845 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.612851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.614284 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.614286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.615598 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.615604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.617025 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.617028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.617181 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.617205 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.617284 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.617347 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":53,"spinlock_wait_cycles":3200,"thread_start_us":78,"threads_started":1}
I20260927 16:32:08.617538 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.611735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.15.93.129:40607)
0927 16:32:08.612114 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.612128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.612151 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:32:08.612227 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.612232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.612410 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.612561 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:08.612966 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.614047 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.615718 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.616690 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.616783 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.617178 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.617320 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.617403 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":73,"threads_started":1}
I20260927 16:32:08.617823 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.610801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:46811 (local address 127.0.0.1:51408)
0927 16:32:08.611469 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.611518 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.611653 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.612528 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.612536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.612561 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:08.612906 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.612913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.614049 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.614051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.615543 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.615554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.616916 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.616919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.617020 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.617068 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.617666 (+ 598us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.617733 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":105,"spinlock_wait_cycles":896,"thread_start_us":75,"threads_started":1}
I20260927 16:32:08.619904 1767 tablet_service.cc:1467] Processing CreateTablet for tablet 5439659cd1914bf494227ae4829bb2a5 (DEFAULT_TABLE table=test-table [id=f592d08caa154006b6f25e4d7922a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:08.620335 1499 tablet_service.cc:1467] Processing CreateTablet for tablet 5439659cd1914bf494227ae4829bb2a5 (DEFAULT_TABLE table=test-table [id=f592d08caa154006b6f25e4d7922a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:08.620334 1633 tablet_service.cc:1467] Processing CreateTablet for tablet 5439659cd1914bf494227ae4829bb2a5 (DEFAULT_TABLE table=test-table [id=f592d08caa154006b6f25e4d7922a4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:08.620846 1767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5439659cd1914bf494227ae4829bb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:08.621294 1633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5439659cd1914bf494227ae4829bb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:08.621294 1499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5439659cd1914bf494227ae4829bb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:08.626470 1850 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Bootstrap starting.
I20260927 16:32:08.627890 1851 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Bootstrap starting.
I20260927 16:32:08.628603 1852 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Bootstrap starting.
I20260927 16:32:08.629949 1850 tablet_bootstrap.cc:654] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:08.629940 1851 tablet_bootstrap.cc:654] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:08.630332 1852 tablet_bootstrap.cc:654] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:08.631289 1851 log.cc:824] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:08.631289 1850 log.cc:824] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:08.631457 1852 log.cc:824] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:08.633136 1851 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: No bootstrap required, opened a new log
I20260927 16:32:08.633315 1850 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: No bootstrap required, opened a new log
I20260927 16:32:08.633315 1852 tablet_bootstrap.cc:492] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: No bootstrap required, opened a new log
I20260927 16:32:08.633335 1851 ts_tablet_manager.cc:1397] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:08.633489 1850 ts_tablet_manager.cc:1397] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 16:32:08.633489 1852 ts_tablet_manager.cc:1397] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:32:08.636210 1851 raft_consensus.cc:348] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.636435 1851 raft_consensus.cc:374] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:08.636487 1851 raft_consensus.cc:729] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2873b44f28417fb5f33885028b0389, State: Initialized, Role: FOLLOWER
I20260927 16:32:08.636904 1851 consensus_queue.cc:260] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.637298 1852 raft_consensus.cc:348] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.637532 1852 raft_consensus.cc:374] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:08.637533 1699 heartbeater.cc:503] Master 127.15.93.190:38687 was elected leader, sending a full tablet report...
I20260927 16:32:08.637599 1852 raft_consensus.cc:729] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180f4270d5104f069beff891795c860d, State: Initialized, Role: FOLLOWER
I20260927 16:32:08.638002 1851 ts_tablet_manager.cc:1428] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:32:08.638003 1852 consensus_queue.cc:260] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.637298 1850 raft_consensus.cc:348] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.638667 1850 raft_consensus.cc:374] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:08.638727 1850 raft_consensus.cc:729] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48c9d8dd2814db3a08341a7fb97ccbf, State: Initialized, Role: FOLLOWER
I20260927 16:32:08.638885 1852 ts_tablet_manager.cc:1428] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:32:08.639106 1850 consensus_queue.cc:260] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.639215 1565 heartbeater.cc:503] Master 127.15.93.190:38687 was elected leader, sending a full tablet report...
I20260927 16:32:08.639905 1850 ts_tablet_manager.cc:1428] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 16:32:08.640139 1833 heartbeater.cc:503] Master 127.15.93.190:38687 was elected leader, sending a full tablet report...
W20260927 16:32:08.810828 1834 tablet.cc:2369] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:32:08.838048 1700 tablet.cc:2369] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 16:32:08.877327 1566 tablet.cc:2369] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:08.958990 1857 raft_consensus.cc:482] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:08.959270 1857 raft_consensus.cc:504] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.960587 1857 leader_election.cc:288] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b2873b44f28417fb5f33885028b0389 (127.15.93.130:46811), e48c9d8dd2814db3a08341a7fb97ccbf (127.15.93.131:37815)
I20260927 16:32:08.964685 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.960934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:37815 (local address 127.15.93.129:49585)
0927 16:32:08.961303 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.961319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.961433 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.961696 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.961699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.961721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:08.961828 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.961837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.962602 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.962605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.963570 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.963579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.964224 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.964228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.964333 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.964376 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.964436 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.964517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":27,"thread_start_us":93,"threads_started":1}
I20260927 16:32:08.964694 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.961033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:49585 (local address 127.15.93.131:37815)
0927 16:32:08.961219 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.961226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.961428 (+ 202us) server_negotiation.cc:407] Connection header received
0927 16:32:08.961513 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.961515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.961584 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.961665 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:08.961971 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.962486 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.963692 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.964096 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.964224 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.964277 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.964478 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.964561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 16:32:08.964753 1846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.960939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47033 (local address 127.15.93.130:46811)
0927 16:32:08.961095 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:08.961099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:08.961285 (+ 186us) server_negotiation.cc:407] Connection header received
0927 16:32:08.961415 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:08.961418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:08.961480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:08.961563 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:08.961870 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.962396 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.963735 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.964108 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.964240 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.964293 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:08.964549 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 16:32:08.964629 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:32:08.964730 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:08.960804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:46811 (local address 127.15.93.129:47033)
0927 16:32:08.961171 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:08.961187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:08.961309 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:08.961584 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:08.961587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:08.961606 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:08.961745 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.961753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.962520 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.962526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.963594 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:08.963603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:08.964243 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:08.964247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:08.964396 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:08.964414 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:32:08.964530 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 16:32:08.964594 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":59,"thread_start_us":112,"threads_started":1}
I20260927 16:32:08.965498 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5439659cd1914bf494227ae4829bb2a5" candidate_uuid: "180f4270d5104f069beff891795c860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" is_pre_election: true
I20260927 16:32:08.965493 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5439659cd1914bf494227ae4829bb2a5" candidate_uuid: "180f4270d5104f069beff891795c860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2873b44f28417fb5f33885028b0389" is_pre_election: true
I20260927 16:32:08.965780 1787 raft_consensus.cc:2454] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 180f4270d5104f069beff891795c860d in term 0.
I20260927 16:32:08.965785 1653 raft_consensus.cc:2454] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 180f4270d5104f069beff891795c860d in term 0.
I20260927 16:32:08.966293 1452 leader_election.cc:302] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 180f4270d5104f069beff891795c860d, 3b2873b44f28417fb5f33885028b0389, e48c9d8dd2814db3a08341a7fb97ccbf; no voters:
I20260927 16:32:08.966580 1857 raft_consensus.cc:2781] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:08.966671 1857 raft_consensus.cc:482] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:08.966713 1857 raft_consensus.cc:3037] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:08.967753 1857 raft_consensus.cc:504] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.968086 1857 leader_election.cc:288] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [CANDIDATE]: Term 1 election: Requested vote from peers 3b2873b44f28417fb5f33885028b0389 (127.15.93.130:46811), e48c9d8dd2814db3a08341a7fb97ccbf (127.15.93.131:37815)
I20260927 16:32:08.968425 1653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5439659cd1914bf494227ae4829bb2a5" candidate_uuid: "180f4270d5104f069beff891795c860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b2873b44f28417fb5f33885028b0389"
I20260927 16:32:08.968449 1787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5439659cd1914bf494227ae4829bb2a5" candidate_uuid: "180f4270d5104f069beff891795c860d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf"
I20260927 16:32:08.968549 1653 raft_consensus.cc:3037] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:08.968564 1787 raft_consensus.cc:3037] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:08.969538 1787 raft_consensus.cc:2454] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180f4270d5104f069beff891795c860d in term 1.
I20260927 16:32:08.969542 1653 raft_consensus.cc:2454] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180f4270d5104f069beff891795c860d in term 1.
I20260927 16:32:08.969841 1452 leader_election.cc:302] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180f4270d5104f069beff891795c860d, e48c9d8dd2814db3a08341a7fb97ccbf; no voters:
I20260927 16:32:08.970018 1857 raft_consensus.cc:2781] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:08.970263 1857 raft_consensus.cc:686] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 1 LEADER]: Becoming Leader. State: Replica: 180f4270d5104f069beff891795c860d, State: Running, Role: LEADER
I20260927 16:32:08.970589 1857 consensus_queue.cc:237] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } }
I20260927 16:32:08.973677 1373 catalog_manager.cc:5495] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d reported cstate change: term changed from 0 to 1, leader changed from <none> to 180f4270d5104f069beff891795c860d (127.15.93.129). New cstate: current_term: 1 leader_uuid: "180f4270d5104f069beff891795c860d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180f4270d5104f069beff891795c860d" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 40607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 } health_report { overall_health: UNKNOWN } } }
I20260927 16:32:09.073416 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.069474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.15.93.129:40607)
0927 16:32:09.069596 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.069601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.069615 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:32:09.069742 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.069745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.069805 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.069926 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:09.070206 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.070722 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.071862 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.072103 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.072222 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.072556 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:09.072977 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:09.073146 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.073229 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.073271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 16:32:09.079836 1787 raft_consensus.cc:1261] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 1 FOLLOWER]: Refusing update from remote peer 180f4270d5104f069beff891795c860d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:09.079826 1653 raft_consensus.cc:1261] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 1 FOLLOWER]: Refusing update from remote peer 180f4270d5104f069beff891795c860d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:09.080581 1861 consensus_queue.cc:1035] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48c9d8dd2814db3a08341a7fb97ccbf" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:09.080824 1857 consensus_queue.cc:1035] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b2873b44f28417fb5f33885028b0389" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:09.088049 1866 mvcc.cc:205] Tried to move back new op lower bound from 7333997482302234624 to 7333997481865891840. Current Snapshot: MvccSnapshot[applied={T|T < 7333997482302234624}]
I20260927 16:32:09.095346 1374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:09.095543 1374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:09.098119 1374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b0aac07e9a04dc4a6a93b84268e2ec6: Sending DeleteTablet for 3 replicas of tablet 5439659cd1914bf494227ae4829bb2a5
I20260927 16:32:09.098963 1633 tablet_service.cc:1514] Processing DeleteTablet for tablet 5439659cd1914bf494227ae4829bb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:09 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:51408
I20260927 16:32:09.099108 1499 tablet_service.cc:1514] Processing DeleteTablet for tablet 5439659cd1914bf494227ae4829bb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:09 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:41144
I20260927 16:32:09.099541 1875 tablet_replica.cc:330] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d: stopping tablet replica
I20260927 16:32:09.099545 1767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5439659cd1914bf494227ae4829bb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:09 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:59792
I20260927 16:32:09.099875 1875 raft_consensus.cc:2229] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:32:09.100244 1874 tablet_replica.cc:330] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: stopping tablet replica
I20260927 16:32:09.100243 1875 raft_consensus.cc:2258] T 5439659cd1914bf494227ae4829bb2a5 P 180f4270d5104f069beff891795c860d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:09.100543 1874 raft_consensus.cc:2229] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:09.100838 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1436
I20260927 16:32:09.100859 1874 raft_consensus.cc:2258] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:09.100046 1876 tablet_replica.cc:330] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: stopping tablet replica
I20260927 16:32:09.103080 1874 ts_tablet_manager.cc:1905] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:09.103355 1876 raft_consensus.cc:2229] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:09.103678 1876 raft_consensus.cc:2258] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:09.104827 1876 ts_tablet_manager.cc:1905] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:09.106564 1874 ts_tablet_manager.cc:1918] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:09.106673 1874 log.cc:1196] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/5439659cd1914bf494227ae4829bb2a5
I20260927 16:32:09.107009 1874 ts_tablet_manager.cc:1939] T 5439659cd1914bf494227ae4829bb2a5 P 3b2873b44f28417fb5f33885028b0389: Deleting consensus metadata
I20260927 16:32:09.107007 1876 ts_tablet_manager.cc:1918] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:09.107116 1876 log.cc:1196] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/5439659cd1914bf494227ae4829bb2a5
I20260927 16:32:09.107456 1876 ts_tablet_manager.cc:1939] T 5439659cd1914bf494227ae4829bb2a5 P e48c9d8dd2814db3a08341a7fb97ccbf: Deleting consensus metadata
I20260927 16:32:09.108428 1360 catalog_manager.cc:4841] TS 3b2873b44f28417fb5f33885028b0389 (127.15.93.130:46811): tablet 5439659cd1914bf494227ae4829bb2a5 (table test-table [id=f592d08caa154006b6f25e4d7922a4ad]) successfully deleted
I20260927 16:32:09.108489 1358 catalog_manager.cc:4841] TS e48c9d8dd2814db3a08341a7fb97ccbf (127.15.93.131:37815): tablet 5439659cd1914bf494227ae4829bb2a5 (table test-table [id=f592d08caa154006b6f25e4d7922a4ad]) successfully deleted
W20260927 16:32:09.109067 1359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:40607 (error 108)
I20260927 16:32:09.109258 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1570
I20260927 16:32:09.110630 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.110281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:40607 (local address 127.0.0.1:41164)
0927 16:32:09.110441 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.110527 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 16:32:09.110806 1359 catalog_manager.cc:4584] TS 180f4270d5104f069beff891795c860d (127.15.93.129:40607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5439659cd1914bf494227ae4829bb2a5: Network error: Client connection negotiation failed: client connection to 127.15.93.129:40607: connect: Connection refused (error 111)
I20260927 16:32:09.115522 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1704
I20260927 16:32:09.121182 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1342
2026-09-27T16:32:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2606 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T16:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:09Z Disabled control of system clock
I20260927 16:32:09.151262 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36507
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:09.262785 1881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:09.263047 1881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:09.263082 1881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:09.267215 1881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:09.267274 1881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:09.267289 1881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:09.267305 1881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:09.267321 1881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:09.272089 1881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:09.273082 1881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:09.274379 1881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:09.282505 1887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.282513 1889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.282502 1886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:09.282799 1881 server_base.cc:1034] running on GCE node
I20260927 16:32:09.283429 1881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:09.284425 1881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:09.285583 1881 hybrid_clock.cc:648] HybridClock initialized: now 1790526729285566 us; error 30 us; skew 500 ppm
I20260927 16:32:09.287276 1881 webserver.cc:460] Webserver started at http://127.15.93.190:40711/ using document root <none> and password file <none>
I20260927 16:32:09.287858 1881 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:09.287917 1881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:09.288071 1881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:09.289963 1881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "f3745c90f3c8421da2d11ad010a5e116"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "f5cfa8c97df0bb56f1383565bd5b0602"
server_key_iv: "33e7e5fc90474e58ac90fe4d789ef5e4"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.290467 1881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "f3745c90f3c8421da2d11ad010a5e116"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "f5cfa8c97df0bb56f1383565bd5b0602"
server_key_iv: "33e7e5fc90474e58ac90fe4d789ef5e4"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.293937 1881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:32:09.296370 1895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.297562 1881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:09.297715 1881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "f3745c90f3c8421da2d11ad010a5e116"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "f5cfa8c97df0bb56f1383565bd5b0602"
server_key_iv: "33e7e5fc90474e58ac90fe4d789ef5e4"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.297829 1881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:09.311851 1881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:09.312520 1881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:09.312721 1881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:09.319844 1947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36507 every 8 connection(s)
I20260927 16:32:09.319847 1881 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36507
I20260927 16:32:09.320825 1881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:09.323899 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:09.327023 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1881
I20260927 16:32:09.327207 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:09.327548 15734 external_mini_cluster.cc:1428] Setting key dfe582e357da917cdb121f4f97712c28
I20260927 16:32:09.329810 1948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: Bootstrap starting.
I20260927 16:32:09.332446 1948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:09.333272 1948 log.cc:824] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:09.335479 1948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: No bootstrap required, opened a new log
I20260927 16:32:09.338196 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.329026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.15.93.190:36507)
0927 16:32:09.329697 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.329717 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.329757 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:32:09.330573 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.330596 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.331106 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.331474 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:09.331908 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.333384 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.334452 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.335118 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.335279 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.336161 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:09.336167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:09.336183 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:09.336679 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:09.336749 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:09.336774 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:09.336866 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.336971 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.337252 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260927 16:32:09.340070 1948 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } }
I20260927 16:32:09.340353 1948 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:09.340413 1948 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3745c90f3c8421da2d11ad010a5e116, State: Initialized, Role: FOLLOWER
I20260927 16:32:09.340890 1948 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } }
I20260927 16:32:09.341041 1948 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:09.341107 1948 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:09.341228 1948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:09.342273 1948 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } }
I20260927 16:32:09.342626 1948 leader_election.cc:302] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3745c90f3c8421da2d11ad010a5e116; no voters:
I20260927 16:32:09.342975 1948 leader_election.cc:288] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:09.343097 1953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:09.343346 1953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [term 1 LEADER]: Becoming Leader. State: Replica: f3745c90f3c8421da2d11ad010a5e116, State: Running, Role: LEADER
I20260927 16:32:09.343703 1953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } }
I20260927 16:32:09.344244 1948 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:09.345430 1955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3745c90f3c8421da2d11ad010a5e116. Latest consensus state: current_term: 1 leader_uuid: "f3745c90f3c8421da2d11ad010a5e116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } } }
I20260927 16:32:09.345388 1954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3745c90f3c8421da2d11ad010a5e116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3745c90f3c8421da2d11ad010a5e116" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36507 } } }
I20260927 16:32:09.345537 1954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:09.345537 1955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:09.345983 1962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:09.349210 1962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:09.354648 1962 catalog_manager.cc:1347] Generated new cluster ID: 3671b1ad5fee4fe8872db7bdfc14d0f2
I20260927 16:32:09.354730 1962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:09.369710 1962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:09.370669 1962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:09.378146 1962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: Generated new TSK 0
I20260927 16:32:09.378836 1962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:09.387998 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--builtin_ntp_servers=127.15.93.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:09.497851 1972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:09.498131 1972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:09.498198 1972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:09.502249 1972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:09.502319 1972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:09.502405 1972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:09.507055 1972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:09.508122 1972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:09.509466 1972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:09.518939 1977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.518956 1978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.518939 1980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:09.519224 1972 server_base.cc:1034] running on GCE node
I20260927 16:32:09.519696 1972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:09.520314 1972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:09.521482 1972 hybrid_clock.cc:648] HybridClock initialized: now 1790526729521440 us; error 46 us; skew 500 ppm
I20260927 16:32:09.523295 1972 webserver.cc:460] Webserver started at http://127.15.93.129:41705/ using document root <none> and password file <none>
I20260927 16:32:09.523900 1972 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:09.523962 1972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:09.524189 1972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:09.526187 1972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "81d4c45d262f4a1681f4dab5f411806f"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "72e18e313c8fcf6d0993610b689bf493"
server_key_iv: "7cd4ef193a3bdd8b3edf8997f05c9b55"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.526738 1972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "81d4c45d262f4a1681f4dab5f411806f"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "72e18e313c8fcf6d0993610b689bf493"
server_key_iv: "7cd4ef193a3bdd8b3edf8997f05c9b55"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.530342 1972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 16:32:09.532718 1986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.533885 1972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:09.534001 1972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "81d4c45d262f4a1681f4dab5f411806f"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "72e18e313c8fcf6d0993610b689bf493"
server_key_iv: "7cd4ef193a3bdd8b3edf8997f05c9b55"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.534106 1972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:09.552646 1972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:09.553375 1972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:09.553574 1972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:09.554171 1972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:09.555136 1972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:09.555208 1972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.555279 1972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:09.555323 1972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.564033 1972 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:43855
I20260927 16:32:09.564110 2099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:43855 every 8 connection(s)
I20260927 16:32:09.565025 1972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:09.573607 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.566749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:56805 (local address 127.15.93.190:36507)
0927 16:32:09.566911 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.566917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.567938 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 16:32:09.569253 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.569255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.569305 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.569378 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:09.570223 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.570743 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.571928 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.572156 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.572237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.573212 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:09.573215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:09.573217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:09.573228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:09.573247 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:09.573254 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:09.573311 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.573442 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.573490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 16:32:09.574450 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.567173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36507 (local address 127.15.93.129:56805)
0927 16:32:09.567788 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.567826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.569054 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.569540 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.569546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.569628 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:09.570103 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.570115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.570847 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.570850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.571817 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.571827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.572284 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.572287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.572417 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.572969 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:09.572971 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:09.573009 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:09.573069 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:09.573325 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:09.573331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:09.573335 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.573424 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.573629 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260927 16:32:09.574713 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 1972
I20260927 16:32:09.574862 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:09.575161 15734 external_mini_cluster.cc:1428] Setting key 58cba41b16a5e54723b94b2142b1deb9
I20260927 16:32:09.575866 2100 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36507
I20260927 16:32:09.576136 2100 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:09.576721 2100 heartbeater.cc:511] Master 127.15.93.190:36507 requested a full tablet report, sending...
I20260927 16:32:09.577330 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--builtin_ntp_servers=127.15.93.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:09.578171 1912 ts_manager.cc:194] Registered new tserver with Master: 81d4c45d262f4a1681f4dab5f411806f (127.15.93.129:43855)
I20260927 16:32:09.579866 1912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.129:56805
W20260927 16:32:09.688711 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:09.688956 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:09.688993 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:09.692817 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:09.692862 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:09.692921 2103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:09.697413 2103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:09.698359 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:09.699669 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:09.708875 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.708860 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:09.708983 2103 server_base.cc:1034] running on GCE node
W20260927 16:32:09.708845 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:09.709520 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:09.710151 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:09.711306 2103 hybrid_clock.cc:648] HybridClock initialized: now 1790526729711293 us; error 24 us; skew 500 ppm
I20260927 16:32:09.713037 2103 webserver.cc:460] Webserver started at http://127.15.93.130:43099/ using document root <none> and password file <none>
I20260927 16:32:09.713614 2103 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:09.713675 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:09.713915 2103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:09.715968 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "d734a97e4e25434a8d09594d62f07a62"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "4393d3bf31bdce55a528e2fce96c66d3"
server_key_iv: "327d6c5cddf43e74a40357369c511504"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.716511 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "d734a97e4e25434a8d09594d62f07a62"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "4393d3bf31bdce55a528e2fce96c66d3"
server_key_iv: "327d6c5cddf43e74a40357369c511504"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.720077 2103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:09.722558 2117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.723881 2103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:09.724007 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "d734a97e4e25434a8d09594d62f07a62"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "4393d3bf31bdce55a528e2fce96c66d3"
server_key_iv: "327d6c5cddf43e74a40357369c511504"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.724116 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:09.741395 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:09.742069 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:09.742275 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:09.742843 2103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:09.743870 2103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:09.743940 2103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.744010 2103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:09.744062 2103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.752774 2103 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:35617
I20260927 16:32:09.752830 2230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:35617 every 8 connection(s)
I20260927 16:32:09.753695 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:09.762653 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.755521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:60493 (local address 127.15.93.190:36507)
0927 16:32:09.755669 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.755672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.756711 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 16:32:09.758034 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.758036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.758081 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.758162 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:09.758981 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.759566 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.760675 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.760879 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.760953 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.762028 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:09.762031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:09.762033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:09.762043 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:09.762059 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:09.762065 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:09.762133 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.762357 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.762436 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:32:09.763605 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.755949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36507 (local address 127.15.93.130:60493)
0927 16:32:09.756588 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.756629 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.757850 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.758322 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.758328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.758408 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:09.758865 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.758876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.759678 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.759681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.760558 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.760567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.761000 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.761003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.761162 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.761806 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:09.761808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:09.761846 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:09.761904 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:09.762170 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:09.762181 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:09.762187 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.762318 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.762639 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260927 16:32:09.763945 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2103
I20260927 16:32:09.764083 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:09.764463 15734 external_mini_cluster.cc:1428] Setting key 69b9f9951b97e47f8f02c8d6c3464cf9
I20260927 16:32:09.764837 2231 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36507
I20260927 16:32:09.765107 2231 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:09.765652 2231 heartbeater.cc:511] Master 127.15.93.190:36507 requested a full tablet report, sending...
I20260927 16:32:09.766475 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--builtin_ntp_servers=127.15.93.148:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:09.766705 1912 ts_manager.cc:194] Registered new tserver with Master: d734a97e4e25434a8d09594d62f07a62 (127.15.93.130:35617)
I20260927 16:32:09.767521 1912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.130:60493
W20260927 16:32:09.877430 2234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:09.877723 2234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:09.877794 2234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:09.881937 2234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:09.882019 2234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:09.882117 2234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:09.886829 2234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:36507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:09.888056 2234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:09.889398 2234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:09.899058 2242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.899067 2240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:09.899075 2239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:09.899669 2234 server_base.cc:1034] running on GCE node
I20260927 16:32:09.900081 2234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:09.900731 2234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:09.901917 2234 hybrid_clock.cc:648] HybridClock initialized: now 1790526729901900 us; error 38 us; skew 500 ppm
I20260927 16:32:09.903849 2234 webserver.cc:460] Webserver started at http://127.15.93.131:40929/ using document root <none> and password file <none>
I20260927 16:32:09.904412 2234 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:09.904501 2234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:09.904732 2234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:09.906836 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "13c57948c98c4a7db2c3ad04e9ab12dd"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "de94fd235ffac2ac2882a6d76e44f2a6"
server_key_iv: "dab6f344adfc1a77a996cdc5662c016e"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.907542 2234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "13c57948c98c4a7db2c3ad04e9ab12dd"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "de94fd235ffac2ac2882a6d76e44f2a6"
server_key_iv: "dab6f344adfc1a77a996cdc5662c016e"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.911048 2234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:32:09.913508 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.914723 2234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:09.914858 2234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "13c57948c98c4a7db2c3ad04e9ab12dd"
format_stamp: "Formatted at 2026-09-27 16:32:09 on dist-test-slave-grzf"
server_key: "de94fd235ffac2ac2882a6d76e44f2a6"
server_key_iv: "dab6f344adfc1a77a996cdc5662c016e"
server_key_version: "encryptionkey@0"
I20260927 16:32:09.914973 2234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:09.933234 2234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:09.933908 2234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:09.934106 2234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:09.934684 2234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:09.935599 2234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:09.935668 2234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.935736 2234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:09.935786 2234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:09.944769 2234 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:40859
I20260927 16:32:09.944809 2361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:40859 every 8 connection(s)
I20260927 16:32:09.945858 2234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:32:09.953778 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2234
I20260927 16:32:09.953914 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:09.954212 15734 external_mini_cluster.cc:1428] Setting key f4bed70975d0e88602a88cfd446ed88c
I20260927 16:32:09.955015 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.947828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:49555 (local address 127.15.93.190:36507)
0927 16:32:09.947964 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.947968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.949076 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 16:32:09.950404 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.950407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.950454 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.950535 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:09.951436 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.951983 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.953066 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.953285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.953363 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.954523 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:09.954527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:09.954529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:09.954549 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:09.954566 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:09.954574 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:09.954658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.954817 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.954878 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:32:09.955941 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.948263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36507 (local address 127.15.93.131:49555)
0927 16:32:09.948909 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.948948 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.950188 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.950691 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.950698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.950779 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:09.951268 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.951280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.952100 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.952103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.952963 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.952972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.953411 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.953413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.953543 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.954214 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:09.954216 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:09.954255 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:09.954346 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:09.954659 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:09.954671 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:09.954679 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.954814 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.955081 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260927 16:32:09.956959 2362 heartbeater.cc:347] Connected to a master server at 127.15.93.190:36507
I20260927 16:32:09.957197 2362 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:09.957687 2362 heartbeater.cc:511] Master 127.15.93.190:36507 requested a full tablet report, sending...
I20260927 16:32:09.958652 1912 ts_manager.cc:194] Registered new tserver with Master: 13c57948c98c4a7db2c3ad04e9ab12dd (127.15.93.131:40859)
I20260927 16:32:09.959211 1912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.131:49555
I20260927 16:32:09.968660 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:09.974402 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.970708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.15.93.190:36507)
0927 16:32:09.970853 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.970857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.970937 (+ 80us) server_negotiation.cc:407] Connection header received
0927 16:32:09.971027 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.971029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.971068 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.971133 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:09.971340 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.972120 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.972910 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.973117 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.973193 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.973914 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:09.973917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:09.973919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:09.973930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:09.973947 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:09.973954 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:09.974051 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.974173 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.974233 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 16:32:09.977180 1912 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:09.980155 1912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:09.994518 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.988592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35617 (local address 127.0.0.1:33522)
0927 16:32:09.989084 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.989101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.989179 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.990182 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.990188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.990211 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:09.990365 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.990373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.991609 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.991612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.992925 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.992935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.993941 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.993945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.994055 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.994092 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.994217 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.994332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260927 16:32:09.994566 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.989163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43855 (local address 127.0.0.1:59386)
0927 16:32:09.989375 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.989390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.989482 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.990340 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.990343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.990360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:09.990476 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.990486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.991958 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.991962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.993194 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.993201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.994219 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.994223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.994299 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.994317 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.994387 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.994459 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":106,"thread_start_us":101,"threads_started":1}
I20260927 16:32:09.994776 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.989568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.15.93.129:43855)
0927 16:32:09.989894 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.989904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.989926 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:32:09.990000 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.990008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.990204 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.990358 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:09.990608 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.991827 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.993310 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.993962 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.994061 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.994231 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.994460 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.994565 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
I20260927 16:32:09.995117 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.989066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.15.93.130:35617)
0927 16:32:09.989606 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.989618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.989642 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:32:09.989718 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.989725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.989936 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.990504 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:09.990630 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.991492 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.993057 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.993826 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.993949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.994144 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.994862 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.994963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":101,"threads_started":1}
I20260927 16:32:09.995865 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.988393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:40859 (local address 127.0.0.1:58730)
0927 16:32:09.988924 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:09.988974 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:09.989094 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:09.990548 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:09.990550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:09.990567 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:09.990665 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.990673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.992049 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.992052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.993528 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:09.993538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.995464 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.995468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:09.995559 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.995582 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:32:09.995657 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 16:32:09.995726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":22,"thread_start_us":83,"threads_started":1}
I20260927 16:32:09.996372 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:09.988527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.15.93.131:40859)
0927 16:32:09.990085 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:09.990097 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:09.990120 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:32:09.990197 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:09.990201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:09.990375 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:09.990568 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:09.990778 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.991893 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.993655 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:09.994435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:09.994536 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:09.995145 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:09.996124 (+ 979us) server_negotiation.cc:299] Negotiation successful
0927 16:32:09.996217 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":91,"threads_started":1}
I20260927 16:32:09.997591 2165 tablet_service.cc:1467] Processing CreateTablet for tablet b7ced612936d48ad842202955754ef35 (DEFAULT_TABLE table=test-table [id=d73edc48b2814ac5a4d5e66f987bcc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:09.997582 2034 tablet_service.cc:1467] Processing CreateTablet for tablet b7ced612936d48ad842202955754ef35 (DEFAULT_TABLE table=test-table [id=d73edc48b2814ac5a4d5e66f987bcc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:09.998502 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ced612936d48ad842202955754ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:09.998508 2034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ced612936d48ad842202955754ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:10.002379 2296 tablet_service.cc:1467] Processing CreateTablet for tablet b7ced612936d48ad842202955754ef35 (DEFAULT_TABLE table=test-table [id=d73edc48b2814ac5a4d5e66f987bcc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:10.003439 2296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ced612936d48ad842202955754ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:10.005549 2380 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: Bootstrap starting.
I20260927 16:32:10.006358 2379 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Bootstrap starting.
I20260927 16:32:10.007613 2380 tablet_bootstrap.cc:654] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:10.008309 2380 log.cc:824] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:10.008477 2379 tablet_bootstrap.cc:654] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:10.009280 2379 log.cc:824] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:10.010344 2380 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: No bootstrap required, opened a new log
I20260927 16:32:10.010541 2380 ts_tablet_manager.cc:1397] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:32:10.011430 2379 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: No bootstrap required, opened a new log
I20260927 16:32:10.011616 2379 ts_tablet_manager.cc:1397] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 16:32:10.011713 2383 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Bootstrap starting.
I20260927 16:32:10.013794 2383 tablet_bootstrap.cc:654] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:10.014081 2380 raft_consensus.cc:348] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.014312 2380 raft_consensus.cc:374] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:10.014361 2380 raft_consensus.cc:729] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d734a97e4e25434a8d09594d62f07a62, State: Initialized, Role: FOLLOWER
I20260927 16:32:10.014600 2383 log.cc:824] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:10.014851 2380 consensus_queue.cc:260] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.015784 2380 ts_tablet_manager.cc:1428] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:32:10.015816 2379 raft_consensus.cc:348] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.016050 2379 raft_consensus.cc:374] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:10.016117 2379 raft_consensus.cc:729] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d4c45d262f4a1681f4dab5f411806f, State: Initialized, Role: FOLLOWER
I20260927 16:32:10.016533 2383 tablet_bootstrap.cc:492] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: No bootstrap required, opened a new log
I20260927 16:32:10.016515 2379 consensus_queue.cc:260] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.016713 2383 ts_tablet_manager.cc:1397] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:32:10.017480 2379 ts_tablet_manager.cc:1428] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:32:10.017835 2100 heartbeater.cc:503] Master 127.15.93.190:36507 was elected leader, sending a full tablet report...
I20260927 16:32:10.019641 2231 heartbeater.cc:503] Master 127.15.93.190:36507 was elected leader, sending a full tablet report...
I20260927 16:32:10.020617 2383 raft_consensus.cc:348] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.020870 2383 raft_consensus.cc:374] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:10.020929 2383 raft_consensus.cc:729] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c57948c98c4a7db2c3ad04e9ab12dd, State: Initialized, Role: FOLLOWER
I20260927 16:32:10.021337 2383 consensus_queue.cc:260] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.021970 2362 heartbeater.cc:503] Master 127.15.93.190:36507 was elected leader, sending a full tablet report...
I20260927 16:32:10.022303 2383 ts_tablet_manager.cc:1428] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:32:10.065692 2387 raft_consensus.cc:482] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:10.065959 2387 raft_consensus.cc:504] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
W20260927 16:32:10.066095 2101 tablet.cc:2369] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:10.067258 2387 leader_election.cc:288] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d734a97e4e25434a8d09594d62f07a62 (127.15.93.130:35617), 81d4c45d262f4a1681f4dab5f411806f (127.15.93.129:43855)
I20260927 16:32:10.071576 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.067628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:54947 (local address 127.15.93.129:43855)
0927 16:32:10.067750 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:10.067754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:10.068024 (+ 270us) server_negotiation.cc:407] Connection header received
0927 16:32:10.068181 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:10.068184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:10.068237 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:10.068315 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:10.068608 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.069081 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.070453 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.071048 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.071163 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:10.071206 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:10.071408 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:32:10.071474 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:32:10.071570 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.067651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:51037 (local address 127.15.93.130:35617)
0927 16:32:10.067789 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:10.067796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:10.067812 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:32:10.067862 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:10.067864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:10.067903 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:10.067985 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:10.068266 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.068818 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.070338 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.070952 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.071074 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:10.071145 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:10.071352 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 16:32:10.071447 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:32:10.071560 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.067447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35617 (local address 127.15.93.131:51037)
0927 16:32:10.067605 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:10.067622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:10.067711 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:10.068009 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:10.068012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:10.068030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:10.068153 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:10.068158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.068927 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.068932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:10.070220 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:10.070230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.071068 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.071071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:10.071192 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:10.071217 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:32:10.071298 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:32:10.071406 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":3,"spinlock_wait_cycles":3328}
I20260927 16:32:10.071563 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.067521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43855 (local address 127.15.93.131:54947)
0927 16:32:10.067917 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:10.067928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:10.068058 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:10.068333 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:10.068336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:10.068351 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:10.068476 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:10.068498 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.069196 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.069199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:10.070351 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:10.070359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.071164 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.071168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:10.071265 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:10.071296 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 16:32:10.071359 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 16:32:10.071428 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":145,"threads_started":1}
I20260927 16:32:10.072328 2185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ced612936d48ad842202955754ef35" candidate_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d734a97e4e25434a8d09594d62f07a62" is_pre_election: true
I20260927 16:32:10.072561 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ced612936d48ad842202955754ef35" candidate_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d4c45d262f4a1681f4dab5f411806f" is_pre_election: true
I20260927 16:32:10.072630 2185 raft_consensus.cc:2454] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c57948c98c4a7db2c3ad04e9ab12dd in term 0.
I20260927 16:32:10.072851 2054 raft_consensus.cc:2454] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c57948c98c4a7db2c3ad04e9ab12dd in term 0.
I20260927 16:32:10.073164 2250 leader_election.cc:302] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c57948c98c4a7db2c3ad04e9ab12dd, d734a97e4e25434a8d09594d62f07a62; no voters:
I20260927 16:32:10.073431 2387 raft_consensus.cc:2781] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:10.073510 2387 raft_consensus.cc:482] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:10.073553 2387 raft_consensus.cc:3037] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:10.074561 2387 raft_consensus.cc:504] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.074923 2387 leader_election.cc:288] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [CANDIDATE]: Term 1 election: Requested vote from peers d734a97e4e25434a8d09594d62f07a62 (127.15.93.130:35617), 81d4c45d262f4a1681f4dab5f411806f (127.15.93.129:43855)
I20260927 16:32:10.075264 2185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ced612936d48ad842202955754ef35" candidate_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d734a97e4e25434a8d09594d62f07a62"
I20260927 16:32:10.075259 2054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ced612936d48ad842202955754ef35" candidate_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d4c45d262f4a1681f4dab5f411806f"
I20260927 16:32:10.075419 2054 raft_consensus.cc:3037] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:10.075408 2185 raft_consensus.cc:3037] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:10.076644 2185 raft_consensus.cc:2454] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c57948c98c4a7db2c3ad04e9ab12dd in term 1.
I20260927 16:32:10.076644 2054 raft_consensus.cc:2454] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c57948c98c4a7db2c3ad04e9ab12dd in term 1.
I20260927 16:32:10.076927 2250 leader_election.cc:302] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c57948c98c4a7db2c3ad04e9ab12dd, d734a97e4e25434a8d09594d62f07a62; no voters:
I20260927 16:32:10.077087 2387 raft_consensus.cc:2781] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:10.077294 2387 raft_consensus.cc:686] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 1 LEADER]: Becoming Leader. State: Replica: 13c57948c98c4a7db2c3ad04e9ab12dd, State: Running, Role: LEADER
I20260927 16:32:10.077622 2387 consensus_queue.cc:237] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } }
I20260927 16:32:10.080787 1911 catalog_manager.cc:5495] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c57948c98c4a7db2c3ad04e9ab12dd (127.15.93.131). New cstate: current_term: 1 leader_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c57948c98c4a7db2c3ad04e9ab12dd" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 40859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 } health_report { overall_health: UNKNOWN } } }
I20260927 16:32:10.093160 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.089962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.15.93.131:40859)
0927 16:32:10.090076 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:10.090085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:10.090097 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:32:10.090204 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:10.090206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:10.090255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:10.090338 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:10.090541 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.091030 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.091891 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:10.092094 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:10.092169 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:10.092433 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:10.092828 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:10.092938 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:10.093030 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 16:32:10.093060 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:32:10.098552 2054 raft_consensus.cc:1261] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 1 FOLLOWER]: Refusing update from remote peer 13c57948c98c4a7db2c3ad04e9ab12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:10.098562 2185 raft_consensus.cc:1261] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 1 FOLLOWER]: Refusing update from remote peer 13c57948c98c4a7db2c3ad04e9ab12dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:10.099282 2387 consensus_queue.cc:1035] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d4c45d262f4a1681f4dab5f411806f" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:10.099555 2389 consensus_queue.cc:1035] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d734a97e4e25434a8d09594d62f07a62" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 35617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:10.105916 2392 mvcc.cc:205] Tried to move back new op lower bound from 7333997486477201408 to 7333997486400536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333997486477201408}]
I20260927 16:32:10.106565 2394 mvcc.cc:205] Tried to move back new op lower bound from 7333997486477201408 to 7333997486400536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333997486477201408}]
I20260927 16:32:10.108032 2393 mvcc.cc:205] Tried to move back new op lower bound from 7333997486477201408 to 7333997486400536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333997486477201408}]
I20260927 16:32:10.115499 1911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:10.115692 1911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:10.118116 1911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3745c90f3c8421da2d11ad010a5e116: Sending DeleteTablet for 3 replicas of tablet b7ced612936d48ad842202955754ef35
I20260927 16:32:10.118923 2165 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ced612936d48ad842202955754ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:10 UTC) from {username='slave'} at 127.0.0.1:33522
I20260927 16:32:10.119067 2296 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ced612936d48ad842202955754ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:10 UTC) from {username='slave'} at 127.0.0.1:58730
I20260927 16:32:10.119086 2034 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ced612936d48ad842202955754ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:10 UTC) from {username='slave'} at 127.0.0.1:59386
I20260927 16:32:10.119620 2404 tablet_replica.cc:330] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f: stopping tablet replica
I20260927 16:32:10.119639 2402 tablet_replica.cc:330] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62: stopping tablet replica
I20260927 16:32:10.119810 2403 tablet_replica.cc:330] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: stopping tablet replica
I20260927 16:32:10.119956 2402 raft_consensus.cc:2229] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:10.120025 2403 raft_consensus.cc:2229] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:32:10.120070 2404 raft_consensus.cc:2229] T b7ced612936d48ad842202955754ef35 P 81d4c45d262f4a1681f4dab5f411806f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:10.120186 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1972
I20260927 16:32:10.121369 2403 raft_consensus.cc:2258] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:10.122970 2403 ts_tablet_manager.cc:1905] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:10.125645 2403 ts_tablet_manager.cc:1918] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:10.125741 2403 log.cc:1196] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/b7ced612936d48ad842202955754ef35
I20260927 16:32:10.125907 2402 raft_consensus.cc:2258] T b7ced612936d48ad842202955754ef35 P d734a97e4e25434a8d09594d62f07a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:10.126049 2403 ts_tablet_manager.cc:1939] T b7ced612936d48ad842202955754ef35 P 13c57948c98c4a7db2c3ad04e9ab12dd: Deleting consensus metadata
W20260927 16:32:10.126873 1899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:43855 (error 108)
I20260927 16:32:10.127158 1898 catalog_manager.cc:4841] TS 13c57948c98c4a7db2c3ad04e9ab12dd (127.15.93.131:40859): tablet b7ced612936d48ad842202955754ef35 (table test-table [id=d73edc48b2814ac5a4d5e66f987bcc88]) successfully deleted
I20260927 16:32:10.127671 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2103
I20260927 16:32:10.128793 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.128274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:43855 (local address 127.0.0.1:59392)
0927 16:32:10.128617 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:10.128694 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:43855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
W20260927 16:32:10.129155 1899 catalog_manager.cc:4584] TS 81d4c45d262f4a1681f4dab5f411806f (127.15.93.129:43855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7ced612936d48ad842202955754ef35: Network error: Client connection negotiation failed: client connection to 127.15.93.129:43855: connect: Connection refused (error 111)
I20260927 16:32:10.133882 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2234
I20260927 16:32:10.134775 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:10.134087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:35617 (local address 127.0.0.1:33526)
0927 16:32:10.134626 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:10.134686 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.130:35617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443}
I20260927 16:32:10.139508 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 1881
2026-09-27T16:32:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:10 dist-test-slave-grzf krb5kdc[2408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:10 dist-test-slave-grzf krb5kdc[2408](info): set up 2 sockets
Sep 27 16:32:10 dist-test-slave-grzf krb5kdc[2408](info): commencing operation
krb5kdc: starting...
W20260927 16:32:11.179428 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:11.296139 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:37877
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:11.405160 2421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:11.405395 2421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:11.405433 2421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:11.409437 2421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:11.409492 2421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:11.409511 2421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:11.409528 2421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:11.409543 2421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:11.414669 2421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:37877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:11.415692 2421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:11.416640 2421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:11.424643 2426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:11.424657 2429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:11.424657 2427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:11.424996 2421 server_base.cc:1034] running on GCE node
I20260927 16:32:11.425599 2421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:11.426641 2421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:11.427794 2421 hybrid_clock.cc:648] HybridClock initialized: now 1790526731427779 us; error 30 us; skew 500 ppm
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:11.430284 2421 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:11.431077 2421 webserver.cc:460] Webserver started at http://127.15.93.190:33035/ using document root <none> and password file <none>
I20260927 16:32:11.431665 2421 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:11.431754 2421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:11.431975 2421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:11.433876 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "23995be04ad443c9a6feae2fe5e8e747"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "a96d0ee3ac1b8619509ff31ffa45fcca"
server_key_iv: "2effbf4081346391c0ed629cda802ffa"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.434403 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "23995be04ad443c9a6feae2fe5e8e747"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "a96d0ee3ac1b8619509ff31ffa45fcca"
server_key_iv: "2effbf4081346391c0ed629cda802ffa"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.437966 2421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:32:11.440258 2436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.441443 2421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:11.441574 2421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "23995be04ad443c9a6feae2fe5e8e747"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "a96d0ee3ac1b8619509ff31ffa45fcca"
server_key_iv: "2effbf4081346391c0ed629cda802ffa"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.441682 2421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:11.459856 2421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:11.462559 2421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:11.462762 2421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:11.469606 2421 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:37877
I20260927 16:32:11.469635 2488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:37877 every 8 connection(s)
I20260927 16:32:11.470538 2421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:11.471899 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2421
I20260927 16:32:11.472035 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:11.472374 15734 external_mini_cluster.cc:1428] Setting key 834724c98631ac337ab5d935d06fd6e0
I20260927 16:32:11.473630 2489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:11.479010 2489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: Bootstrap starting.
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:11.481407 2489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:11.482120 2489 log.cc:824] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:11.483933 2489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: No bootstrap required, opened a new log
I20260927 16:32:11.486110 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:11.473739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.15.93.190:37877)
0927 16:32:11.474177 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:11.474192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:11.474226 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:32:11.474865 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:11.474881 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:11.475191 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:11.475556 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:11.476362 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.477359 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.478222 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.478690 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.478863 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:11.481553 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:11.481558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:11.481571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:11.481599 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:11.483338 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.483843 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.483849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.483854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.483966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.484305 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.484308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.484310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.484684 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:11.484889 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:11.485153 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 16:32:11.485431 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260927 16:32:11.487560 2489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } }
I20260927 16:32:11.487808 2489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:11.487862 2489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23995be04ad443c9a6feae2fe5e8e747, State: Initialized, Role: FOLLOWER
I20260927 16:32:11.488309 2489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } }
I20260927 16:32:11.488456 2489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:11.488502 2489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:11.488569 2489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:11.489347 2489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } }
I20260927 16:32:11.489617 2489 leader_election.cc:302] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23995be04ad443c9a6feae2fe5e8e747; no voters:
I20260927 16:32:11.489858 2489 leader_election.cc:288] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:11.490043 2494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:11.490345 2494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [term 1 LEADER]: Becoming Leader. State: Replica: 23995be04ad443c9a6feae2fe5e8e747, State: Running, Role: LEADER
I20260927 16:32:11.490643 2494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } }
I20260927 16:32:11.491014 2489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:11.492168 2496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23995be04ad443c9a6feae2fe5e8e747. Latest consensus state: current_term: 1 leader_uuid: "23995be04ad443c9a6feae2fe5e8e747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } } }
I20260927 16:32:11.492306 2496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:11.492426 2495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23995be04ad443c9a6feae2fe5e8e747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23995be04ad443c9a6feae2fe5e8e747" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 37877 } } }
I20260927 16:32:11.492535 2495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:11.493057 2503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:11.495642 2503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:11.500463 2503 catalog_manager.cc:1347] Generated new cluster ID: 260d7cdaa6bc4bcdb346f64391b3599c
I20260927 16:32:11.500526 2503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:11.520539 2503 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:11.521796 2503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:11.529038 2503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: Generated new TSK 0
I20260927 16:32:11.529793 2503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.93.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
Entry for principal kudu/127.15.93.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab.
I20260927 16:32:11.564414 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:11.675941 2515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:11.676203 2515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:11.676245 2515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:11.680248 2515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:11.680303 2515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:11.680397 2515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:11.685557 2515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.129.keytab
--principal=kudu/127.15.93.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:11.686589 2515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:11.687652 2515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:11.696987 2520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:11.697034 2521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:11.697135 2515 server_base.cc:1034] running on GCE node
W20260927 16:32:11.697032 2523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:11.697693 2515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:11.698315 2515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:11.699471 2515 hybrid_clock.cc:648] HybridClock initialized: now 1790526731699452 us; error 24 us; skew 500 ppm
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:11.702262 2515 init.cc:377] Logged in from keytab as kudu/127.15.93.129@KRBTEST.COM (short username kudu)
I20260927 16:32:11.703114 2515 webserver.cc:460] Webserver started at http://127.15.93.129:42249/ using document root <none> and password file <none>
I20260927 16:32:11.703740 2515 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:11.703835 2515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:11.704058 2515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:11.706010 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "efafd5d4f32842fc81b50db90f65ac53"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "ecd0200c5240a25f80c85e43fd070423"
server_key_iv: "a5c6d26439b9f4e83af429d9a265fdce"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.706562 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "efafd5d4f32842fc81b50db90f65ac53"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "ecd0200c5240a25f80c85e43fd070423"
server_key_iv: "a5c6d26439b9f4e83af429d9a265fdce"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.710116 2515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:32:11.712414 2530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.713577 2515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:11.713714 2515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "efafd5d4f32842fc81b50db90f65ac53"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "ecd0200c5240a25f80c85e43fd070423"
server_key_iv: "a5c6d26439b9f4e83af429d9a265fdce"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.713822 2515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:11.730093 2515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:11.733364 2515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:11.733572 2515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:11.734150 2515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:11.734972 2515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:11.735040 2515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.735110 2515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:11.735152 2515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.744242 2515 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:41667
I20260927 16:32:11.744271 2643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:41667 every 8 connection(s)
I20260927 16:32:11.745217 2515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:11.750420 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2515
I20260927 16:32:11.750578 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:11.750954 15734 external_mini_cluster.cc:1428] Setting key c6fa0a26786a8875aae27469d72d2e09
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.129@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.93.130@KRBTEST.COM; defaulting to no policy
I20260927 16:32:11.760237 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:11.747078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:43291 (local address 127.15.93.190:37877)
0927 16:32:11.747251 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:11.747254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:11.748175 (+ 921us) server_negotiation.cc:407] Connection header received
0927 16:32:11.749610 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:11.749614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:11.749668 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:11.749748 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:11.751427 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.751977 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.753275 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.753531 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.753646 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:11.756251 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:11.756255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:11.756258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:11.756288 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:11.757794 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.758301 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.758306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.758308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.758388 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.758851 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.758857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.758860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.759128 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:11.759249 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:11.759930 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 16:32:11.760073 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 16:32:11.761786 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:11.747553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37877 (local address 127.15.93.129:43291)
0927 16:32:11.748015 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:11.748058 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:11.749367 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:11.749969 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:11.749976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:11.750527 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:11.751228 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:11.751244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.752109 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.752114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:11.753141 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:11.753151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.753681 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.753686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:11.753865 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:11.754497 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:11.754513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:11.756091 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:11.757913 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:11.757919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:11.757935 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:11.758194 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:11.758492 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:11.758495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:11.758497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:11.758679 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:11.759273 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:11.759281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:11.759645 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 16:32:11.760400 (+ 755us) client_negotiation.cc:240] Negotiation successful
0927 16:32:11.760756 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.130@KRBTEST.COM" created.
I20260927 16:32:11.769415 2644 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37877
I20260927 16:32:11.769702 2644 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:11.770355 2644 heartbeater.cc:511] Master 127.15.93.190:37877 requested a full tablet report, sending...
I20260927 16:32:11.772272 2453 ts_manager.cc:194] Registered new tserver with Master: efafd5d4f32842fc81b50db90f65ac53 (127.15.93.129:41667)
I20260927 16:32:11.773514 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.129@KRBTEST.COM'} at 127.15.93.129:43291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
Entry for principal kudu/127.15.93.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab.
I20260927 16:32:11.779835 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:11.888643 2649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:11.888900 2649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:11.888943 2649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:11.893190 2649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:11.893261 2649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:11.893347 2649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:11.898558 2649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.130.keytab
--principal=kudu/127.15.93.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:11.899652 2649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:11.900707 2649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:32:11.910475 2649 server_base.cc:1034] running on GCE node
W20260927 16:32:11.910593 2654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:11.910677 2657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:11.910641 2655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:11.911227 2649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:11.911963 2649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:11.913151 2649 hybrid_clock.cc:648] HybridClock initialized: now 1790526731913133 us; error 28 us; skew 500 ppm
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:11.916013 2649 init.cc:377] Logged in from keytab as kudu/127.15.93.130@KRBTEST.COM (short username kudu)
I20260927 16:32:11.916894 2649 webserver.cc:460] Webserver started at http://127.15.93.130:41001/ using document root <none> and password file <none>
I20260927 16:32:11.917418 2649 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:11.917475 2649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:11.917697 2649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:11.919796 2649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "dcdb1d2da4964ae3912866fcd0510319"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "19f858a96880352ba6370149151edbae"
server_key_iv: "08534f1c9f2936e5c79a0e37b8c5bd82"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.920297 2649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "dcdb1d2da4964ae3912866fcd0510319"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "19f858a96880352ba6370149151edbae"
server_key_iv: "08534f1c9f2936e5c79a0e37b8c5bd82"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.923904 2649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:32:11.926172 2664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.927502 2649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:11.927625 2649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "dcdb1d2da4964ae3912866fcd0510319"
format_stamp: "Formatted at 2026-09-27 16:32:11 on dist-test-slave-grzf"
server_key: "19f858a96880352ba6370149151edbae"
server_key_iv: "08534f1c9f2936e5c79a0e37b8c5bd82"
server_key_version: "encryptionkey@0"
I20260927 16:32:11.927726 2649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:11.963742 2649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:11.966651 2649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:11.966850 2649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:11.967494 2649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:11.968315 2649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:11.968360 2649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.968422 2649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:11.968454 2649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:11.977694 2649 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:34359
I20260927 16:32:11.977718 2777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:34359 every 8 connection(s)
I20260927 16:32:11.978576 2649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:11.986130 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2649
I20260927 16:32:11.986310 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:11.986688 15734 external_mini_cluster.cc:1428] Setting key 33d2728342aa1f018c1d2b633f34f184
Sep 27 16:32:11 dist-test-slave-grzf krb5kdc[2408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526731, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.130@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:11.992718 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:11.980250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:40725 (local address 127.15.93.190:37877)
0927 16:32:11.980398 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:11.980403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:11.981303 (+ 900us) server_negotiation.cc:407] Connection header received
0927 16:32:11.982559 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:11.982562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:11.982609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:11.982687 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:11.983914 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.984430 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.985523 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.985750 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.985861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:11.988650 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:11.988653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:11.988655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:11.988677 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:11.990257 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.990744 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.990749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.990752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.990884 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:11.991259 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:11.991265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:11.991269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:11.991543 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:11.991652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:11.992298 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 16:32:11.992482 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 16:32:11.994004 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:11.980699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37877 (local address 127.15.93.130:40725)
0927 16:32:11.981152 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:11.981194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:11.982353 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:11.982872 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:11.982882 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:11.983252 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:11.983791 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:11.983803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.984533 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.984535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:11.985390 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:11.985398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:11.985920 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:11.985924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:11.986075 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:11.986687 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:11.986702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:11.988459 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:11.990376 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:11.990381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:11.990393 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:11.990646 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:11.991010 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:11.991012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:11.991014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:11.991156 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:11.991681 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:11.991686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:11.992035 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 16:32:11.992217 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 16:32:11.992534 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260927 16:32:11.995227 2778 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37877
I20260927 16:32:11.995520 2778 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:11.996037 2778 heartbeater.cc:511] Master 127.15.93.190:37877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.93.131@KRBTEST.COM; defaulting to no policy
I20260927 16:32:11.997114 2453 ts_manager.cc:194] Registered new tserver with Master: dcdb1d2da4964ae3912866fcd0510319 (127.15.93.130:34359)
I20260927 16:32:11.997684 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.130@KRBTEST.COM'} at 127.15.93.130:40725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
Entry for principal kudu/127.15.93.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab.
I20260927 16:32:12.014644 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:12.123025 2783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:12.123272 2783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:12.123314 2783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:12.127269 2783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:12.127329 2783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:12.127404 2783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:12.132937 2783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.131.keytab
--principal=kudu/127.15.93.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:37877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:12.133976 2783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:12.134999 2783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:12.146240 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:12.146245 2789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:12.146241 2791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:12.147121 2783 server_base.cc:1034] running on GCE node
I20260927 16:32:12.147621 2783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:12.148463 2783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:12.149618 2783 hybrid_clock.cc:648] HybridClock initialized: now 1790526732149589 us; error 42 us; skew 500 ppm
Sep 27 16:32:12 dist-test-slave-grzf krb5kdc[2408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:12.152456 2783 init.cc:377] Logged in from keytab as kudu/127.15.93.131@KRBTEST.COM (short username kudu)
I20260927 16:32:12.153337 2783 webserver.cc:460] Webserver started at http://127.15.93.131:37615/ using document root <none> and password file <none>
I20260927 16:32:12.153851 2783 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:12.153908 2783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:12.154080 2783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:12.156188 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "ec226776851b44c997fcfb17a74f2ba7"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "6c4fe6b6819c0daa5ba0102b29d49b21"
server_key_iv: "4a74c2c27a1a03fcd992a11aed98f536"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.156699 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "ec226776851b44c997fcfb17a74f2ba7"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "6c4fe6b6819c0daa5ba0102b29d49b21"
server_key_iv: "4a74c2c27a1a03fcd992a11aed98f536"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.160339 2783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:12.162904 2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.164187 2783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:12.164288 2783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "ec226776851b44c997fcfb17a74f2ba7"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "6c4fe6b6819c0daa5ba0102b29d49b21"
server_key_iv: "4a74c2c27a1a03fcd992a11aed98f536"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.164373 2783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:12.177276 2783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:12.180344 2783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:12.180485 2783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:12.180968 2783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:12.181694 2783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:12.181737 2783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.181773 2783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:12.181797 2783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.190157 2783 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:43997
I20260927 16:32:12.190179 2911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:43997 every 8 connection(s)
I20260927 16:32:12.190969 2783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
Sep 27 16:32:12 dist-test-slave-grzf krb5kdc[2408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.131@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:12.200824 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2783
I20260927 16:32:12.200973 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:12.201282 15734 external_mini_cluster.cc:1428] Setting key 4665cc9cabb62780718a3a0103feb10b
I20260927 16:32:12.204406 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.192829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:54143 (local address 127.15.93.190:37877)
0927 16:32:12.192940 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.192944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.193727 (+ 783us) server_negotiation.cc:407] Connection header received
0927 16:32:12.195067 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.195071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.195139 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.195218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:12.196373 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.196909 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.198027 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.198262 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.198373 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.200854 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:12.200857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:12.200859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:12.200916 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:12.202501 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:12.202978 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:12.202982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:12.202984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:12.203073 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:12.203474 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:12.203477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:12.203478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:12.203658 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:12.203734 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.204198 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.204293 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 16:32:12.205156 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.193156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:37877 (local address 127.15.93.131:54143)
0927 16:32:12.193571 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.193628 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.194856 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.195427 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.195435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.195784 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:12.196230 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.196243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.197035 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.197038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.197902 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.197911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.198409 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.198414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.198581 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.199092 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:12.199107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:12.200663 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:12.202606 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:12.202614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:12.202628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:12.202886 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:12.203165 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:12.203170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:12.203175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:12.203343 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:12.203750 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:12.203755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:12.203971 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.204129 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.204366 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260927 16:32:12.206220 2912 heartbeater.cc:347] Connected to a master server at 127.15.93.190:37877
I20260927 16:32:12.206467 2912 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:12.206923 2912 heartbeater.cc:511] Master 127.15.93.190:37877 requested a full tablet report, sending...
I20260927 16:32:12.207885 2453 ts_manager.cc:194] Registered new tserver with Master: ec226776851b44c997fcfb17a74f2ba7 (127.15.93.131:43997)
I20260927 16:32:12.208434 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.93.131@KRBTEST.COM'} at 127.15.93.131:54143
I20260927 16:32:12.215008 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:12.224460 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.216977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.15.93.190:37877)
0927 16:32:12.217103 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.217108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.217307 (+ 199us) server_negotiation.cc:407] Connection header received
0927 16:32:12.217458 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.217461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.217500 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.217563 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:12.218250 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.218858 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.219738 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.219940 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.220021 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.221263 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:12.221266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:12.221268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:12.221289 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:12.222764 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:12.223177 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:12.223180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:12.223182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:12.223261 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:12.223580 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:12.223591 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:12.223593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:12.223801 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:12.223941 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.224185 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.224293 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:32:12.227535 2453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:12.230424 2453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:12.244297 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.239000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.15.93.129:41667)
0927 16:32:12.239792 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.239810 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.239836 (+ 26us) server_negotiation.cc:407] Connection header received
0927 16:32:12.239929 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.239937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.240190 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.240645 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:12.240798 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.241529 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.242960 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.243600 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.243724 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.243873 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.244057 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.244150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":113,"threads_started":1}
I20260927 16:32:12.244287 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.238741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41667 (local address 127.0.0.1:33534)
0927 16:32:12.239430 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.239486 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.239607 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.240362 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.240369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.240387 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:12.240714 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.240719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.241660 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.241666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.242832 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.242841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.243723 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.243726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.243831 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.243867 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.244020 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.244143 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":299,"thread_start_us":106,"threads_started":1}
I20260927 16:32:12.245157 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.239943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34359 (local address 127.0.0.1:32902)
0927 16:32:12.240182 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.240196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.240280 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.240707 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.240710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.240727 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:12.240817 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.240824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.242239 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.242242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.243605 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.243613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.244772 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.244776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.244860 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.244882 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.244967 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.245034 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":143,"thread_start_us":88,"threads_started":1}
I20260927 16:32:12.245630 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.239612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.15.93.130:34359)
0927 16:32:12.240284 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.240295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.240319 (+ 24us) server_negotiation.cc:407] Connection header received
0927 16:32:12.240395 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.240401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.240573 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.240727 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:12.240923 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.242082 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.243743 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.244636 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.244734 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.244869 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.245012 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.245485 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":134,"threads_started":1}
I20260927 16:32:12.245740 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.238741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:43997 (local address 127.0.0.1:35990)
0927 16:32:12.239056 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.239828 (+ 772us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.239909 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.240420 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.240423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.240434 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:12.240610 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.240618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.242239 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.242242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.243602 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.243613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.245213 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.245218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.245427 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.245450 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.245497 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.245557 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":110,"thread_start_us":71,"threads_started":1}
I20260927 16:32:12.246592 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.239358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.15.93.131:43997)
0927 16:32:12.239889 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.239900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.239923 (+ 23us) server_negotiation.cc:407] Connection header received
0927 16:32:12.240039 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.240045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.240256 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.240449 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:12.240954 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.242096 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.243742 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.245078 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.246085 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.246333 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.246383 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.246478 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":93,"threads_started":1}
I20260927 16:32:12.247740 2578 tablet_service.cc:1467] Processing CreateTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 (DEFAULT_TABLE table=test-table [id=90e272c85f41499faa868cacd64add5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:12.248289 2712 tablet_service.cc:1467] Processing CreateTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 (DEFAULT_TABLE table=test-table [id=90e272c85f41499faa868cacd64add5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:12.248862 2578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f87f51fbb4f0fa44e4ec0124f7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:12.248680 2846 tablet_service.cc:1467] Processing CreateTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 (DEFAULT_TABLE table=test-table [id=90e272c85f41499faa868cacd64add5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:12.249290 2712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f87f51fbb4f0fa44e4ec0124f7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:12.249656 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f87f51fbb4f0fa44e4ec0124f7e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:12.255316 2929 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Bootstrap starting.
I20260927 16:32:12.256989 2930 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: Bootstrap starting.
I20260927 16:32:12.257678 2929 tablet_bootstrap.cc:654] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:12.259011 2929 log.cc:824] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:12.259225 2930 tablet_bootstrap.cc:654] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:12.259526 2931 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Bootstrap starting.
I20260927 16:32:12.260540 2930 log.cc:824] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:12.261363 2929 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: No bootstrap required, opened a new log
I20260927 16:32:12.261559 2929 ts_tablet_manager.cc:1397] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 16:32:12.261736 2931 tablet_bootstrap.cc:654] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:12.262514 2930 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: No bootstrap required, opened a new log
I20260927 16:32:12.262683 2930 ts_tablet_manager.cc:1397] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:12.263166 2931 log.cc:824] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:12.264765 2931 tablet_bootstrap.cc:492] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: No bootstrap required, opened a new log
I20260927 16:32:12.264953 2931 ts_tablet_manager.cc:1397] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:12.265443 2929 raft_consensus.cc:348] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.265714 2929 raft_consensus.cc:374] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:12.265769 2929 raft_consensus.cc:729] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec226776851b44c997fcfb17a74f2ba7, State: Initialized, Role: FOLLOWER
I20260927 16:32:12.266219 2929 consensus_queue.cc:260] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.266359 2930 raft_consensus.cc:348] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.266618 2930 raft_consensus.cc:374] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:12.266676 2930 raft_consensus.cc:729] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efafd5d4f32842fc81b50db90f65ac53, State: Initialized, Role: FOLLOWER
I20260927 16:32:12.266984 2912 heartbeater.cc:503] Master 127.15.93.190:37877 was elected leader, sending a full tablet report...
I20260927 16:32:12.267107 2930 consensus_queue.cc:260] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.268090 2930 ts_tablet_manager.cc:1428] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 16:32:12.268090 2929 ts_tablet_manager.cc:1428] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:32:12.268483 2644 heartbeater.cc:503] Master 127.15.93.190:37877 was elected leader, sending a full tablet report...
I20260927 16:32:12.269606 2931 raft_consensus.cc:348] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.269862 2931 raft_consensus.cc:374] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:12.269917 2931 raft_consensus.cc:729] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdb1d2da4964ae3912866fcd0510319, State: Initialized, Role: FOLLOWER
I20260927 16:32:12.270342 2931 consensus_queue.cc:260] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.271198 2931 ts_tablet_manager.cc:1428] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 16:32:12.271522 2778 heartbeater.cc:503] Master 127.15.93.190:37877 was elected leader, sending a full tablet report...
I20260927 16:32:12.347810 2935 raft_consensus.cc:482] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:12.348057 2935 raft_consensus.cc:504] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.349220 2935 leader_election.cc:288] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efafd5d4f32842fc81b50db90f65ac53 (127.15.93.129:41667), dcdb1d2da4964ae3912866fcd0510319 (127.15.93.130:34359)
I20260927 16:32:12.353258 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.349649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:56949 (local address 127.15.93.130:34359)
0927 16:32:12.349775 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.349780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.350108 (+ 328us) server_negotiation.cc:407] Connection header received
0927 16:32:12.350218 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.350221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.350275 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.350368 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:12.350664 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.351141 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352284 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.352694 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352823 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.352876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.353051 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.353128 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:32:12.353258 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.349528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41667 (local address 127.15.93.131:35045)
0927 16:32:12.349771 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.349786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.349906 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.350233 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.350236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.350253 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:12.350393 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.350403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.351236 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.351240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.352195 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.352204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.352847 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.352963 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.352980 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.353076 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.353137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":47}
I20260927 16:32:12.353291 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.349639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:35045 (local address 127.15.93.129:41667)
0927 16:32:12.349767 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.349774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.349866 (+ 92us) server_negotiation.cc:407] Connection header received
0927 16:32:12.350039 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.350044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.350118 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.350199 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:12.350552 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.351122 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352335 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.352710 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.352895 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.353111 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.353187 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 16:32:12.353272 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.349528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:34359 (local address 127.15.93.131:56949)
0927 16:32:12.349954 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.349983 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.350107 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.350389 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.350391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.350405 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:12.350541 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.350547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.351278 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.351281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.352169 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.352178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.352818 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.352822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.352921 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.352946 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.353012 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.353099 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":80,"thread_start_us":137,"threads_started":1}
I20260927 16:32:12.353963 2732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f87f51fbb4f0fa44e4ec0124f7e49" candidate_uuid: "ec226776851b44c997fcfb17a74f2ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdb1d2da4964ae3912866fcd0510319" is_pre_election: true
I20260927 16:32:12.354048 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f87f51fbb4f0fa44e4ec0124f7e49" candidate_uuid: "ec226776851b44c997fcfb17a74f2ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efafd5d4f32842fc81b50db90f65ac53" is_pre_election: true
I20260927 16:32:12.354310 2732 raft_consensus.cc:2454] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec226776851b44c997fcfb17a74f2ba7 in term 0.
I20260927 16:32:12.354334 2598 raft_consensus.cc:2454] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec226776851b44c997fcfb17a74f2ba7 in term 0.
I20260927 16:32:12.354779 2799 leader_election.cc:302] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdb1d2da4964ae3912866fcd0510319, ec226776851b44c997fcfb17a74f2ba7; no voters:
I20260927 16:32:12.355088 2935 raft_consensus.cc:2781] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:12.355168 2935 raft_consensus.cc:482] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:12.355216 2935 raft_consensus.cc:3037] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:12.356318 2935 raft_consensus.cc:504] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.356699 2935 leader_election.cc:288] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [CANDIDATE]: Term 1 election: Requested vote from peers efafd5d4f32842fc81b50db90f65ac53 (127.15.93.129:41667), dcdb1d2da4964ae3912866fcd0510319 (127.15.93.130:34359)
I20260927 16:32:12.356993 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f87f51fbb4f0fa44e4ec0124f7e49" candidate_uuid: "ec226776851b44c997fcfb17a74f2ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efafd5d4f32842fc81b50db90f65ac53"
I20260927 16:32:12.357086 2732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f87f51fbb4f0fa44e4ec0124f7e49" candidate_uuid: "ec226776851b44c997fcfb17a74f2ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdb1d2da4964ae3912866fcd0510319"
I20260927 16:32:12.357147 2598 raft_consensus.cc:3037] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:12.357215 2732 raft_consensus.cc:3037] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:12.358317 2598 raft_consensus.cc:2454] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec226776851b44c997fcfb17a74f2ba7 in term 1.
I20260927 16:32:12.358327 2732 raft_consensus.cc:2454] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec226776851b44c997fcfb17a74f2ba7 in term 1.
I20260927 16:32:12.358608 2799 leader_election.cc:302] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcdb1d2da4964ae3912866fcd0510319, ec226776851b44c997fcfb17a74f2ba7; no voters:
I20260927 16:32:12.358780 2935 raft_consensus.cc:2781] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:12.359011 2935 raft_consensus.cc:686] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 1 LEADER]: Becoming Leader. State: Replica: ec226776851b44c997fcfb17a74f2ba7, State: Running, Role: LEADER
I20260927 16:32:12.359337 2935 consensus_queue.cc:237] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } }
I20260927 16:32:12.362272 2452 catalog_manager.cc:5495] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec226776851b44c997fcfb17a74f2ba7 (127.15.93.131). New cstate: current_term: 1 leader_uuid: "ec226776851b44c997fcfb17a74f2ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec226776851b44c997fcfb17a74f2ba7" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 43997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 } health_report { overall_health: UNKNOWN } } }
I20260927 16:32:12.395689 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.392037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.15.93.131:43997)
0927 16:32:12.392182 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.392186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.392200 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:32:12.392336 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.392339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.392394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.392507 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:12.392746 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.393319 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.394170 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.394393 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.394520 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.394817 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:12.395225 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:12.395365 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.395459 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.395518 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:32:12.401595 2598 raft_consensus.cc:1261] T a52f87f51fbb4f0fa44e4ec0124f7e49 P efafd5d4f32842fc81b50db90f65ac53 [term 1 FOLLOWER]: Refusing update from remote peer ec226776851b44c997fcfb17a74f2ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:12.401666 2732 raft_consensus.cc:1261] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 1 FOLLOWER]: Refusing update from remote peer ec226776851b44c997fcfb17a74f2ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:12.402362 2935 consensus_queue.cc:1035] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efafd5d4f32842fc81b50db90f65ac53" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:12.402562 2939 consensus_queue.cc:1035] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcdb1d2da4964ae3912866fcd0510319" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:12.410995 2943 mvcc.cc:205] Tried to move back new op lower bound from 7333997495909163008 to 7333997495746527232. Current Snapshot: MvccSnapshot[applied={T|T < 7333997495909163008}]
I20260927 16:32:12.419042 2452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:12.419226 2452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:12.421828 2452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23995be04ad443c9a6feae2fe5e8e747: Sending DeleteTablet for 3 replicas of tablet a52f87f51fbb4f0fa44e4ec0124f7e49
I20260927 16:32:12.422667 2578 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:12 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:33534
I20260927 16:32:12.422710 2712 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:12 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:32902
I20260927 16:32:12.423471 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2515
I20260927 16:32:12.423488 2953 tablet_replica.cc:330] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: stopping tablet replica
I20260927 16:32:12.423480 2846 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f87f51fbb4f0fa44e4ec0124f7e49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:12 UTC) from {username='kudu', principal='kudu/127.15.93.190@KRBTEST.COM'} at 127.0.0.1:35990
I20260927 16:32:12.423811 2953 raft_consensus.cc:2229] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:12.424119 2953 raft_consensus.cc:2258] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:12.426247 2953 ts_tablet_manager.cc:1905] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:12.427588 2954 tablet_replica.cc:330] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: stopping tablet replica
I20260927 16:32:12.427937 2954 raft_consensus.cc:2229] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:32:12.428395 2954 raft_consensus.cc:2258] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:12.428820 2953 ts_tablet_manager.cc:1918] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:12.428910 2953 log.cc:1196] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/a52f87f51fbb4f0fa44e4ec0124f7e49
I20260927 16:32:12.429226 2953 ts_tablet_manager.cc:1939] T a52f87f51fbb4f0fa44e4ec0124f7e49 P dcdb1d2da4964ae3912866fcd0510319: Deleting consensus metadata
I20260927 16:32:12.430078 2954 ts_tablet_manager.cc:1905] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:12.430418 2437 catalog_manager.cc:4841] TS dcdb1d2da4964ae3912866fcd0510319 (127.15.93.130:34359): tablet a52f87f51fbb4f0fa44e4ec0124f7e49 (table test-table [id=90e272c85f41499faa868cacd64add5b]) successfully deleted
W20260927 16:32:12.432133 2440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:41667 (error 108)
I20260927 16:32:12.432680 2954 ts_tablet_manager.cc:1918] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:12.432776 2954 log.cc:1196] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/a52f87f51fbb4f0fa44e4ec0124f7e49
I20260927 16:32:12.432888 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2649
I20260927 16:32:12.433156 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.432854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:41667 (local address 127.0.0.1:33074)
0927 16:32:12.432989 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.433061 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 16:32:12.433421 2440 catalog_manager.cc:4584] TS efafd5d4f32842fc81b50db90f65ac53 (127.15.93.129:41667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a52f87f51fbb4f0fa44e4ec0124f7e49: Network error: Client connection negotiation failed: client connection to 127.15.93.129:41667: connect: Connection refused (error 111)
I20260927 16:32:12.433787 2954 ts_tablet_manager.cc:1939] T a52f87f51fbb4f0fa44e4ec0124f7e49 P ec226776851b44c997fcfb17a74f2ba7: Deleting consensus metadata
I20260927 16:32:12.434764 2437 catalog_manager.cc:4841] TS ec226776851b44c997fcfb17a74f2ba7 (127.15.93.131:43997): tablet a52f87f51fbb4f0fa44e4ec0124f7e49 (table test-table [id=90e272c85f41499faa868cacd64add5b]) successfully deleted
I20260927 16:32:12.439824 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2783
I20260927 16:32:12.444895 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2421
2026-09-27T16:32:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2303 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T16:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:12Z Disabled control of system clock
I20260927 16:32:12.474891 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41243
--webserver_interface=127.15.93.190
--webserver_port=0
--builtin_ntp_servers=127.15.93.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:12.582664 2959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:12.582921 2959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:12.582955 2959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:12.586831 2959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:12.586885 2959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:12.586900 2959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:12.586917 2959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:12.586932 2959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:12.592288 2959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:12.593287 2959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:12.594601 2959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:12.603837 2967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:12.603821 2964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:12.603809 2965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:12.603860 2959 server_base.cc:1034] running on GCE node
I20260927 16:32:12.604652 2959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:12.605737 2959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:12.607052 2959 hybrid_clock.cc:648] HybridClock initialized: now 1790526732607037 us; error 38 us; skew 500 ppm
I20260927 16:32:12.608879 2959 webserver.cc:460] Webserver started at http://127.15.93.190:36459/ using document root <none> and password file <none>
I20260927 16:32:12.609448 2959 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:12.609568 2959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:12.609786 2959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:12.611862 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "1ecde5e05ead4a428f9ca522d847e19e"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "a25d7848615a0d398b1ca49bc12f1a8f"
server_key_iv: "5b4688c33b298e178ce7d7560a76c94a"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.612424 2959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "1ecde5e05ead4a428f9ca522d847e19e"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "a25d7848615a0d398b1ca49bc12f1a8f"
server_key_iv: "5b4688c33b298e178ce7d7560a76c94a"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.616040 2959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:12.618348 2973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.619613 2959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:12.619746 2959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "1ecde5e05ead4a428f9ca522d847e19e"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "a25d7848615a0d398b1ca49bc12f1a8f"
server_key_iv: "5b4688c33b298e178ce7d7560a76c94a"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.619882 2959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:12.629601 2959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:12.630268 2959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:12.630447 2959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:12.637084 2959 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:41243
I20260927 16:32:12.637082 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:41243 every 8 connection(s)
I20260927 16:32:12.637955 2959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:12.640202 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 2959
I20260927 16:32:12.640326 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:12.640640 15734 external_mini_cluster.cc:1428] Setting key 887752624b702713a1368eb1eb0530a5
I20260927 16:32:12.641002 3026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:12.647017 3026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: Bootstrap starting.
I20260927 16:32:12.649170 3026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:12.649724 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.641957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.15.93.190:41243)
0927 16:32:12.642607 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.642622 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.642655 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:12.643624 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.643642 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.644029 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.644350 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:12.644763 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.645686 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.646570 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.647026 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.647192 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.648051 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:12.648056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:12.648067 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:12.648420 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:12.648472 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:12.648490 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:12.648604 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.648718 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.648957 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260927 16:32:12.649952 3026 log.cc:824] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:12.651805 3026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: No bootstrap required, opened a new log
I20260927 16:32:12.654978 3026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } }
I20260927 16:32:12.655197 3026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:12.655277 3026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecde5e05ead4a428f9ca522d847e19e, State: Initialized, Role: FOLLOWER
I20260927 16:32:12.655702 3026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } }
I20260927 16:32:12.655864 3026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:12.655951 3026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:12.656042 3026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:12.656839 3026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } }
I20260927 16:32:12.657188 3026 leader_election.cc:302] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ecde5e05ead4a428f9ca522d847e19e; no voters:
I20260927 16:32:12.657516 3026 leader_election.cc:288] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:12.657615 3031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:12.657786 3031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [term 1 LEADER]: Becoming Leader. State: Replica: 1ecde5e05ead4a428f9ca522d847e19e, State: Running, Role: LEADER
I20260927 16:32:12.658049 3031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } }
I20260927 16:32:12.658694 3026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:12.659605 3033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ecde5e05ead4a428f9ca522d847e19e. Latest consensus state: current_term: 1 leader_uuid: "1ecde5e05ead4a428f9ca522d847e19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } } }
I20260927 16:32:12.659746 3033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:12.660048 3032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ecde5e05ead4a428f9ca522d847e19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ecde5e05ead4a428f9ca522d847e19e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41243 } } }
I20260927 16:32:12.660142 3032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:12.660218 3040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:12.663236 3040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:12.668483 3040 catalog_manager.cc:1347] Generated new cluster ID: a4693a0ac1cf49dfb14d0ac6cffff1dd
I20260927 16:32:12.668545 3040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:12.688699 3040 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:12.689672 3040 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:12.697575 3040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: Generated new TSK 0
I20260927 16:32:12.698361 3040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:12.709486 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.129:0
--local_ip_for_outbound_sockets=127.15.93.129
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--builtin_ntp_servers=127.15.93.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 16:32:12.817900 3050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:12.818145 3050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:12.818181 3050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:12.822193 3050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:12.822245 3050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:12.822330 3050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.129
I20260927 16:32:12.827065 3050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.93.129
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.129
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:12.828115 3050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:12.829882 3050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:12.839051 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:12.839044 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:12.839217 3050 server_base.cc:1034] running on GCE node
W20260927 16:32:12.839007 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:12.839746 3050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:12.840284 3050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:12.841475 3050 hybrid_clock.cc:648] HybridClock initialized: now 1790526732841449 us; error 36 us; skew 500 ppm
I20260927 16:32:12.843175 3050 webserver.cc:460] Webserver started at http://127.15.93.129:38325/ using document root <none> and password file <none>
I20260927 16:32:12.843803 3050 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:12.843894 3050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:12.844105 3050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:12.846053 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data/instance:
uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "c1273ae3afdb0bb2a7a22104995d511d"
server_key_iv: "58cf731e6a4e8e9aa244795fd8f8defd"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.846606 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance:
uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "c1273ae3afdb0bb2a7a22104995d511d"
server_key_iv: "58cf731e6a4e8e9aa244795fd8f8defd"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.850090 3050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 16:32:12.852571 3064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.853716 3050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:12.853843 3050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal
uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8"
format_stamp: "Formatted at 2026-09-27 16:32:12 on dist-test-slave-grzf"
server_key: "c1273ae3afdb0bb2a7a22104995d511d"
server_key_iv: "58cf731e6a4e8e9aa244795fd8f8defd"
server_key_version: "encryptionkey@0"
I20260927 16:32:12.853950 3050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:12.871767 3050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:12.872401 3050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:12.872591 3050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:12.873145 3050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:12.874034 3050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:12.874102 3050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.874168 3050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:12.874219 3050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:12.883378 3050 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.129:44729
I20260927 16:32:12.883431 3177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.129:44729 every 8 connection(s)
I20260927 16:32:12.884264 3050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/data/info.pb
I20260927 16:32:12.885649 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3050
I20260927 16:32:12.885792 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-0/wal/instance
I20260927 16:32:12.886116 15734 external_mini_cluster.cc:1428] Setting key eb0d10c985f121988d880b2eb3777b37
I20260927 16:32:12.888084 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.130:0
--local_ip_for_outbound_sockets=127.15.93.130
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--builtin_ntp_servers=127.15.93.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:12.893565 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.886036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:47261 (local address 127.15.93.190:41243)
0927 16:32:12.886192 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:12.886196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:12.887440 (+ 1244us) server_negotiation.cc:407] Connection header received
0927 16:32:12.888936 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:12.888939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:12.888993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:12.889068 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:12.889934 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.890470 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.891653 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.891883 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.891991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.893061 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:12.893065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:12.893067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:12.893080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:12.893100 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:12.893116 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:12.893191 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:12.893354 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 16:32:12.893428 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:32:12.894346 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:12.886498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41243 (local address 127.15.93.129:47261)
0927 16:32:12.887269 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:12.887319 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:12.888726 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:12.889244 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:12.889251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:12.889335 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:12.889801 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.889814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.890601 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.890606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.891535 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:12.891544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:12.892030 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:12.892034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:12.892197 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:12.892816 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:12.892818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:12.892855 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:12.892914 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:12.893204 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:12.893217 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:12.893226 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 16:32:12.893341 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 16:32:12.893604 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":163,"threads_started":1}
I20260927 16:32:12.895853 3178 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41243
I20260927 16:32:12.896075 3178 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:12.896564 3178 heartbeater.cc:511] Master 127.15.93.190:41243 requested a full tablet report, sending...
I20260927 16:32:12.898016 2990 ts_manager.cc:194] Registered new tserver with Master: f94dd10b9f96460a9f3ccd2d60b3efc8 (127.15.93.129:44729)
I20260927 16:32:12.899148 2990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.129:47261
W20260927 16:32:12.998153 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:12.998399 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:12.998436 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:13.002460 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:13.002512 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:13.002604 3181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.130
I20260927 16:32:13.007344 3181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.93.130
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.130
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:13.008423 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:13.009802 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:32:13.019178 3181 server_base.cc:1034] running on GCE node
W20260927 16:32:13.019133 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:13.019133 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:13.019160 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:13.019877 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:13.020483 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:13.021670 3181 hybrid_clock.cc:648] HybridClock initialized: now 1790526733021636 us; error 60 us; skew 500 ppm
I20260927 16:32:13.023563 3181 webserver.cc:460] Webserver started at http://127.15.93.130:44405/ using document root <none> and password file <none>
I20260927 16:32:13.024279 3181 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:13.024367 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:13.024621 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:13.026583 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data/instance:
uuid: "f83a912fa775492e935b416deeb872aa"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "9f8c11304c6de02f03e943518a13b752"
server_key_iv: "90e9e44abb67a37f6cdb7b2cb6dc0666"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.027135 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance:
uuid: "f83a912fa775492e935b416deeb872aa"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "9f8c11304c6de02f03e943518a13b752"
server_key_iv: "90e9e44abb67a37f6cdb7b2cb6dc0666"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.030805 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:32:13.033142 3195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.034296 3181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:32:13.034440 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal
uuid: "f83a912fa775492e935b416deeb872aa"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "9f8c11304c6de02f03e943518a13b752"
server_key_iv: "90e9e44abb67a37f6cdb7b2cb6dc0666"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.034548 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:13.052696 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:13.053371 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:13.053565 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:13.054160 3181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:13.055069 3181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:13.055140 3181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.055212 3181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:13.055253 3181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.063987 3181 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.130:33429
I20260927 16:32:13.064059 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.130:33429 every 8 connection(s)
I20260927 16:32:13.064862 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/data/info.pb
I20260927 16:32:13.074208 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.066618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.130:37655 (local address 127.15.93.190:41243)
0927 16:32:13.066767 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.066772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.068112 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 16:32:13.069437 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.069440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.069519 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.069602 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:13.070491 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.070998 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.072228 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.072620 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.072707 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.073803 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:13.073805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:13.073807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:13.073820 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:13.073837 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:13.073845 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:13.073903 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.074031 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.074083 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:32:13.074368 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3181
I20260927 16:32:13.074510 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal/instance
I20260927 16:32:13.074826 15734 external_mini_cluster.cc:1428] Setting key b5a63b1a6647ca0529c3697ba0399d78
I20260927 16:32:13.075362 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.067135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41243 (local address 127.15.93.130:37655)
0927 16:32:13.067951 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.067993 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.069236 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.069781 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.069788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.069870 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:13.070353 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.070363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.071116 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.071118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.072117 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.072126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.072775 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.072778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.072932 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.073555 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:13.073556 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:13.073601 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:13.073663 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:13.073917 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:13.073923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:13.073928 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.074023 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.074353 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":122,"threads_started":1}
I20260927 16:32:13.076911 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.93.131:0
--local_ip_for_outbound_sockets=127.15.93.131
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--builtin_ntp_servers=127.15.93.148:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 16:32:13.076989 3309 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41243
I20260927 16:32:13.077212 3309 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:13.077827 3309 heartbeater.cc:511] Master 127.15.93.190:41243 requested a full tablet report, sending...
I20260927 16:32:13.079037 2990 ts_manager.cc:194] Registered new tserver with Master: f83a912fa775492e935b416deeb872aa (127.15.93.130:33429)
I20260927 16:32:13.079901 2990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.130:37655
W20260927 16:32:13.197024 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:13.197264 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:13.197302 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:13.201361 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:13.201408 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:13.201499 3312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.93.131
I20260927 16:32:13.206175 3312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.93.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.93.131
--webserver_port=0
--tserver_master_addrs=127.15.93.190:41243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.93.131
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:13.207178 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:13.208554 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:13.217792 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:13.217854 3312 server_base.cc:1034] running on GCE node
W20260927 16:32:13.217772 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:13.217772 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:13.218448 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:13.219048 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:13.220206 3312 hybrid_clock.cc:648] HybridClock initialized: now 1790526733220194 us; error 25 us; skew 500 ppm
I20260927 16:32:13.222055 3312 webserver.cc:460] Webserver started at http://127.15.93.131:38081/ using document root <none> and password file <none>
I20260927 16:32:13.222630 3312 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:13.222693 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:13.222929 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:13.225021 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data/instance:
uuid: "89dd08b450a7466dbbd1e39061658c1f"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "d3705790c70c7741fc354eedcc1c304d"
server_key_iv: "99b9ed14acf02d7a50314023ba17e988"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.225567 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance:
uuid: "89dd08b450a7466dbbd1e39061658c1f"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "d3705790c70c7741fc354eedcc1c304d"
server_key_iv: "99b9ed14acf02d7a50314023ba17e988"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.229195 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:32:13.231654 3326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.232854 3312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:13.232944 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal
uuid: "89dd08b450a7466dbbd1e39061658c1f"
format_stamp: "Formatted at 2026-09-27 16:32:13 on dist-test-slave-grzf"
server_key: "d3705790c70c7741fc354eedcc1c304d"
server_key_iv: "99b9ed14acf02d7a50314023ba17e988"
server_key_version: "encryptionkey@0"
I20260927 16:32:13.233073 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:13.248394 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:13.249079 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:13.249274 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:13.249867 3312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 16:32:13.250810 3312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 16:32:13.250856 3312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.250922 3312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 16:32:13.250958 3312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:13.259917 3312 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.131:39083
I20260927 16:32:13.260051 3439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.131:39083 every 8 connection(s)
I20260927 16:32:13.260893 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/data/info.pb
I20260927 16:32:13.263783 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3312
I20260927 16:32:13.263891 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal/instance
I20260927 16:32:13.264217 15734 external_mini_cluster.cc:1428] Setting key f95a7dbaed265d6bd61f64c7e6361a67
I20260927 16:32:13.270295 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.262673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.131:34799 (local address 127.15.93.190:41243)
0927 16:32:13.262806 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.262810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.264070 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 16:32:13.265772 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.265774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.265817 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.265874 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:13.266897 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.267506 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.268616 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.268829 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.268906 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.269900 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:13.269903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:13.269905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:13.269916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:13.269934 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:13.269942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:13.269998 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.270138 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.270188 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:32:13.271240 3442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.263091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41243 (local address 127.15.93.131:34799)
0927 16:32:13.263899 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.263956 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.265552 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.266052 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.266061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.266181 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:13.266756 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.266769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.267632 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.267634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.268505 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.268513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.268954 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.268957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.269105 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.269665 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 16:32:13.269667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:13.269705 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 16:32:13.269764 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:13.270014 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:13.270020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:13.270024 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.270117 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.270332 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260927 16:32:13.272313 3440 heartbeater.cc:347] Connected to a master server at 127.15.93.190:41243
I20260927 16:32:13.272517 3440 heartbeater.cc:464] Registering TS with master...
I20260927 16:32:13.273054 3440 heartbeater.cc:511] Master 127.15.93.190:41243 requested a full tablet report, sending...
I20260927 16:32:13.274003 2990 ts_manager.cc:194] Registered new tserver with Master: 89dd08b450a7466dbbd1e39061658c1f (127.15.93.131:39083)
I20260927 16:32:13.274636 2990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.93.131:34799
I20260927 16:32:13.278095 15734 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 16:32:13.283653 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.280098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.15.93.190:41243)
0927 16:32:13.280238 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.280244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.280376 (+ 132us) server_negotiation.cc:407] Connection header received
0927 16:32:13.280476 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.280478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.280514 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.280576 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:13.280832 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.281320 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.282106 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.282305 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.282377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.283092 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:13.283096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:13.283098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:13.283109 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:13.283126 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:13.283134 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:13.283189 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.283284 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.283328 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 16:32:13.286183 2990 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 16:32:13.289237 2990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 16:32:13.303354 3451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.297435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:44729 (local address 127.0.0.1:42144)
0927 16:32:13.298183 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.298197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.298279 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.299214 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.299217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.299232 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:13.299336 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.299342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.300435 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.300440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.301711 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.301722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.302781 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.302785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.302890 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.302935 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.303025 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.303165 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":35,"thread_start_us":114,"threads_started":1}
I20260927 16:32:13.303571 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.298049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:39083 (local address 127.0.0.1:40660)
0927 16:32:13.298506 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.298519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.298613 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.299294 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.299298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.299315 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:13.299439 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.299447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.301067 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.301070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.302141 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.302151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.303183 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.303186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.303276 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.303297 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.303349 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.303453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":85,"threads_started":1}
I20260927 16:32:13.303761 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.297347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:33429 (local address 127.0.0.1:57022)
0927 16:32:13.297787 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.297834 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.297954 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.299040 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.299046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.299068 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:13.299212 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.299218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.300721 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.300724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.301942 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.301950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.303217 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.303221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.303343 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.303363 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.303462 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.303552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":51,"thread_start_us":76,"threads_started":1}
I20260927 16:32:13.304335 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.298125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.15.93.129:44729)
0927 16:32:13.298522 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.298537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.298562 (+ 25us) server_negotiation.cc:407] Connection header received
0927 16:32:13.298640 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.298649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.298865 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.299050 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:13.299478 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.300295 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.301836 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.302645 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.303794 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.304024 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.304076 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.304176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260927 16:32:13.304374 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.298435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.15.93.131:39083)
0927 16:32:13.298849 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.298859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.298881 (+ 22us) server_negotiation.cc:407] Connection header received
0927 16:32:13.298951 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.298958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.299115 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.299294 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:13.299567 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.300946 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.302278 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.303046 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.303936 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.304096 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.304138 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.304218 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":163,"threads_started":1}
I20260927 16:32:13.304853 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.297457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.15.93.130:33429)
0927 16:32:13.298595 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.298609 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.298636 (+ 27us) server_negotiation.cc:407] Connection header received
0927 16:32:13.298712 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.298717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.298877 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.299050 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:13.299338 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.300596 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.302102 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.303064 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.304073 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.304374 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.304429 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.304647 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":84,"threads_started":1}
I20260927 16:32:13.307466 3243 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf84118181447bd841aaf2291fb5f38 (DEFAULT_TABLE table=test-table [id=bdf28d3706494791af62f8f2a6ed9891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:13.307456 3112 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf84118181447bd841aaf2291fb5f38 (DEFAULT_TABLE table=test-table [id=bdf28d3706494791af62f8f2a6ed9891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:13.307524 3374 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf84118181447bd841aaf2291fb5f38 (DEFAULT_TABLE table=test-table [id=bdf28d3706494791af62f8f2a6ed9891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 16:32:13.308516 3243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf84118181447bd841aaf2291fb5f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:13.308524 3374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf84118181447bd841aaf2291fb5f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:13.308524 3112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf84118181447bd841aaf2291fb5f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:13.314880 3457 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Bootstrap starting.
I20260927 16:32:13.316309 3459 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Bootstrap starting.
I20260927 16:32:13.317389 3457 tablet_bootstrap.cc:654] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:13.318135 3457 log.cc:824] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:13.318225 3459 tablet_bootstrap.cc:654] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:13.318900 3458 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Bootstrap starting.
I20260927 16:32:13.318938 3459 log.cc:824] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:13.320300 3457 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: No bootstrap required, opened a new log
I20260927 16:32:13.320500 3457 ts_tablet_manager.cc:1397] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 16:32:13.320851 3458 tablet_bootstrap.cc:654] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:13.321475 3459 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: No bootstrap required, opened a new log
I20260927 16:32:13.321485 3458 log.cc:824] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:13.321657 3459 ts_tablet_manager.cc:1397] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 16:32:13.323467 3458 tablet_bootstrap.cc:492] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: No bootstrap required, opened a new log
I20260927 16:32:13.323674 3458 ts_tablet_manager.cc:1397] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 16:32:13.325340 3457 raft_consensus.cc:348] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.325625 3457 raft_consensus.cc:374] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:13.325510 3459 raft_consensus.cc:348] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.325690 3457 raft_consensus.cc:729] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83a912fa775492e935b416deeb872aa, State: Initialized, Role: FOLLOWER
I20260927 16:32:13.325714 3459 raft_consensus.cc:374] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:13.325758 3459 raft_consensus.cc:729] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94dd10b9f96460a9f3ccd2d60b3efc8, State: Initialized, Role: FOLLOWER
I20260927 16:32:13.326154 3459 consensus_queue.cc:260] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.326154 3457 consensus_queue.cc:260] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.326828 3309 heartbeater.cc:503] Master 127.15.93.190:41243 was elected leader, sending a full tablet report...
I20260927 16:32:13.327086 3178 heartbeater.cc:503] Master 127.15.93.190:41243 was elected leader, sending a full tablet report...
I20260927 16:32:13.327149 3457 ts_tablet_manager.cc:1428] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 16:32:13.327150 3459 ts_tablet_manager.cc:1428] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 16:32:13.328562 3458 raft_consensus.cc:348] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.328785 3458 raft_consensus.cc:374] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:13.328845 3458 raft_consensus.cc:729] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd08b450a7466dbbd1e39061658c1f, State: Initialized, Role: FOLLOWER
I20260927 16:32:13.329285 3458 consensus_queue.cc:260] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.329990 3440 heartbeater.cc:503] Master 127.15.93.190:41243 was elected leader, sending a full tablet report...
I20260927 16:32:13.331879 3458 ts_tablet_manager.cc:1428] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260927 16:32:13.385353 3179 tablet.cc:2369] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:13.392598 3464 raft_consensus.cc:482] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 16:32:13.392799 3464 raft_consensus.cc:504] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.393947 3464 leader_election.cc:288] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83a912fa775492e935b416deeb872aa (127.15.93.130:33429), 89dd08b450a7466dbbd1e39061658c1f (127.15.93.131:39083)
I20260927 16:32:13.398149 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.394346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:53129 (local address 127.15.93.130:33429)
0927 16:32:13.394509 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.394515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.394535 (+ 20us) server_negotiation.cc:407] Connection header received
0927 16:32:13.394787 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.394790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.394835 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.394900 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:13.395224 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.395983 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397225 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.397615 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397749 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.397799 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.397959 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.398030 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:13.398149 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.394250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.130:33429 (local address 127.15.93.129:53129)
0927 16:32:13.394374 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.394390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.394505 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.394947 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.394951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.394974 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:13.395102 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.395110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.396132 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.396135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.397094 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.397110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.397737 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.397826 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.397847 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.397943 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.398014 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":21}
I20260927 16:32:13.398329 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.394682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.93.129:51579 (local address 127.15.93.131:39083)
0927 16:32:13.394839 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.394844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.395079 (+ 235us) server_negotiation.cc:407] Connection header received
0927 16:32:13.395186 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.395190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.395241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.395319 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 16:32:13.395628 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.396276 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397389 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.397788 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397927 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.397972 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.398153 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.398219 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 16:32:13.398345 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.394565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.131:39083 (local address 127.15.93.129:51579)
0927 16:32:13.394993 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.395006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:13.395086 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:13.395342 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:13.395344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:13.395359 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 16:32:13.395512 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.395517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.396379 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.396385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.397264 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:13.397274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.397927 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.397930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:13.398024 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.398044 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 16:32:13.398138 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 16:32:13.398205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":34,"thread_start_us":123,"threads_started":1}
I20260927 16:32:13.398875 3263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf84118181447bd841aaf2291fb5f38" candidate_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83a912fa775492e935b416deeb872aa" is_pre_election: true
I20260927 16:32:13.399080 3394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf84118181447bd841aaf2291fb5f38" candidate_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd08b450a7466dbbd1e39061658c1f" is_pre_election: true
I20260927 16:32:13.399179 3263 raft_consensus.cc:2454] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94dd10b9f96460a9f3ccd2d60b3efc8 in term 0.
I20260927 16:32:13.399422 3394 raft_consensus.cc:2454] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94dd10b9f96460a9f3ccd2d60b3efc8 in term 0.
I20260927 16:32:13.399742 3066 leader_election.cc:302] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f83a912fa775492e935b416deeb872aa, f94dd10b9f96460a9f3ccd2d60b3efc8; no voters:
I20260927 16:32:13.399992 3464 raft_consensus.cc:2781] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 16:32:13.400079 3464 raft_consensus.cc:482] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 16:32:13.400137 3464 raft_consensus.cc:3037] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:13.401134 3464 raft_consensus.cc:504] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.401454 3464 leader_election.cc:288] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [CANDIDATE]: Term 1 election: Requested vote from peers f83a912fa775492e935b416deeb872aa (127.15.93.130:33429), 89dd08b450a7466dbbd1e39061658c1f (127.15.93.131:39083)
I20260927 16:32:13.401871 3394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf84118181447bd841aaf2291fb5f38" candidate_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89dd08b450a7466dbbd1e39061658c1f"
I20260927 16:32:13.401906 3263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf84118181447bd841aaf2291fb5f38" candidate_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83a912fa775492e935b416deeb872aa"
I20260927 16:32:13.402004 3394 raft_consensus.cc:3037] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:13.402045 3263 raft_consensus.cc:3037] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:13.403051 3394 raft_consensus.cc:2454] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94dd10b9f96460a9f3ccd2d60b3efc8 in term 1.
I20260927 16:32:13.403056 3263 raft_consensus.cc:2454] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94dd10b9f96460a9f3ccd2d60b3efc8 in term 1.
I20260927 16:32:13.403340 3066 leader_election.cc:302] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f83a912fa775492e935b416deeb872aa, f94dd10b9f96460a9f3ccd2d60b3efc8; no voters:
I20260927 16:32:13.403589 3464 raft_consensus.cc:2781] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:13.403807 3464 raft_consensus.cc:686] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 1 LEADER]: Becoming Leader. State: Replica: f94dd10b9f96460a9f3ccd2d60b3efc8, State: Running, Role: LEADER
I20260927 16:32:13.404135 3464 consensus_queue.cc:237] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } }
I20260927 16:32:13.407338 2988 catalog_manager.cc:5495] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94dd10b9f96460a9f3ccd2d60b3efc8 (127.15.93.129). New cstate: current_term: 1 leader_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94dd10b9f96460a9f3ccd2d60b3efc8" member_type: VOTER last_known_addr { host: "127.15.93.129" port: 44729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 } health_report { overall_health: UNKNOWN } } }
I20260927 16:32:13.427842 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.424708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.15.93.129:44729)
0927 16:32:13.424841 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:13.424844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:13.424856 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:32:13.424921 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:13.424923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:13.424974 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:13.425061 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:13.425238 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.425713 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.426500 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:13.426702 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:13.426774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:13.427012 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:13.427467 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:13.427557 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:13.427609 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 16:32:13.427653 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 16:32:13.433583 3394 raft_consensus.cc:1261] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 1 FOLLOWER]: Refusing update from remote peer f94dd10b9f96460a9f3ccd2d60b3efc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:13.433601 3263 raft_consensus.cc:1261] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 1 FOLLOWER]: Refusing update from remote peer f94dd10b9f96460a9f3ccd2d60b3efc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:13.434371 3467 consensus_queue.cc:1035] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83a912fa775492e935b416deeb872aa" member_type: VOTER last_known_addr { host: "127.15.93.130" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:13.434585 3464 consensus_queue.cc:1035] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89dd08b450a7466dbbd1e39061658c1f" member_type: VOTER last_known_addr { host: "127.15.93.131" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:13.440865 3470 mvcc.cc:205] Tried to move back new op lower bound from 7333997500136398848 to 7333997500025982976. Current Snapshot: MvccSnapshot[applied={T|T < 7333997500136398848}]
I20260927 16:32:13.442153 3471 mvcc.cc:205] Tried to move back new op lower bound from 7333997500136398848 to 7333997500025982976. Current Snapshot: MvccSnapshot[applied={T|T < 7333997500136398848}]
I20260927 16:32:13.450729 2990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:13.450860 2990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 16:32:13.453123 2990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ecde5e05ead4a428f9ca522d847e19e: Sending DeleteTablet for 3 replicas of tablet 6cf84118181447bd841aaf2291fb5f38
I20260927 16:32:13.453863 3112 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf84118181447bd841aaf2291fb5f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:13 UTC) from {username='slave'} at 127.0.0.1:42144
I20260927 16:32:13.453907 3374 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf84118181447bd841aaf2291fb5f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:13 UTC) from {username='slave'} at 127.0.0.1:40660
I20260927 16:32:13.453903 3243 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf84118181447bd841aaf2291fb5f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 16:32:13 UTC) from {username='slave'} at 127.0.0.1:57022
I20260927 16:32:13.454406 3482 tablet_replica.cc:330] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: stopping tablet replica
I20260927 16:32:13.454562 3480 tablet_replica.cc:330] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8: stopping tablet replica
I20260927 16:32:13.454723 3482 raft_consensus.cc:2229] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:13.454936 3480 raft_consensus.cc:2229] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 16:32:13.455000 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3050
I20260927 16:32:13.455029 3482 raft_consensus.cc:2258] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:13.455438 3480 raft_consensus.cc:2258] T 6cf84118181447bd841aaf2291fb5f38 P f94dd10b9f96460a9f3ccd2d60b3efc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:13.455699 3481 tablet_replica.cc:330] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: stopping tablet replica
I20260927 16:32:13.456017 3481 raft_consensus.cc:2229] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 16:32:13.456349 3481 raft_consensus.cc:2258] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 16:32:13.457197 3482 ts_tablet_manager.cc:1905] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:13.458096 3481 ts_tablet_manager.cc:1905] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 16:32:13.459642 3482 ts_tablet_manager.cc:1918] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:13.459749 3482 log.cc:1196] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-1/wal/wals/6cf84118181447bd841aaf2291fb5f38
I20260927 16:32:13.460041 3482 ts_tablet_manager.cc:1939] T 6cf84118181447bd841aaf2291fb5f38 P f83a912fa775492e935b416deeb872aa: Deleting consensus metadata
I20260927 16:32:13.461159 2975 catalog_manager.cc:4841] TS f83a912fa775492e935b416deeb872aa (127.15.93.130:33429): tablet 6cf84118181447bd841aaf2291fb5f38 (table test-table [id=bdf28d3706494791af62f8f2a6ed9891]) successfully deleted
I20260927 16:32:13.461416 3481 ts_tablet_manager.cc:1918] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 16:32:13.461513 3481 log.cc:1196] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Deleting WAL directory at /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790526488131879-15734-0/minicluster-data/ts-2/wal/wals/6cf84118181447bd841aaf2291fb5f38
I20260927 16:32:13.461802 3481 ts_tablet_manager.cc:1939] T 6cf84118181447bd841aaf2291fb5f38 P 89dd08b450a7466dbbd1e39061658c1f: Deleting consensus metadata
W20260927 16:32:13.463039 2974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.93.129:44729 (error 108)
I20260927 16:32:13.463114 2975 catalog_manager.cc:4841] TS 89dd08b450a7466dbbd1e39061658c1f (127.15.93.131:39083): tablet 6cf84118181447bd841aaf2291fb5f38 (table test-table [id=bdf28d3706494791af62f8f2a6ed9891]) successfully deleted
I20260927 16:32:13.463289 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3181
I20260927 16:32:13.464624 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:13.464275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.129:44729 (local address 127.0.0.1:42168)
0927 16:32:13.464437 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:13.464518 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.129:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 16:32:13.464787 2974 catalog_manager.cc:4584] TS f94dd10b9f96460a9f3ccd2d60b3efc8 (127.15.93.129:44729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cf84118181447bd841aaf2291fb5f38: Network error: Client connection negotiation failed: client connection to 127.15.93.129:44729: connect: Connection refused (error 111)
I20260927 16:32:13.468892 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3312
I20260927 16:32:13.474341 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 2959
2026-09-27T16:32:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1028 ms)
[----------] 6 tests from EncryptionPolicyTest (10185 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:13 dist-test-slave-grzf krb5kdc[3486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:13 dist-test-slave-grzf krb5kdc[3486](info): set up 2 sockets
Sep 27 16:32:13 dist-test-slave-grzf krb5kdc[3486](info): commencing operation
krb5kdc: starting...
W20260927 16:32:14.507544 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:14.627164 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:14.732365 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:14.732650 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:14.732726 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:14.736768 3499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:14.736855 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:14.736892 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:14.736930 3499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:14.736982 3499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:14.741999 3499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:14.742967 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:14.743961 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:14.751971 3507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:14.751956 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:14.751968 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:14.752256 3499 server_base.cc:1034] running on GCE node
I20260927 16:32:14.752904 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:14.753902 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:14.755095 3499 hybrid_clock.cc:648] HybridClock initialized: now 1790526734755069 us; error 43 us; skew 500 ppm
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:14.757802 3499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:14.758607 3499 webserver.cc:460] Webserver started at http://127.0.0.1:44943/ using document root <none> and password file <none>
I20260927 16:32:14.759133 3499 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:14.759222 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:14.759496 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:14.761412 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "6b60c196b383486db4c5d13141d58e9a"
format_stamp: "Formatted at 2026-09-27 16:32:14 on dist-test-slave-grzf"
server_key: "073826a877ffd04a8244afb9fda5a0de"
server_key_iv: "5bd29de94c43328adbfcf6581554aa73"
server_key_version: "encryptionkey@0"
I20260927 16:32:14.761962 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "6b60c196b383486db4c5d13141d58e9a"
format_stamp: "Formatted at 2026-09-27 16:32:14 on dist-test-slave-grzf"
server_key: "073826a877ffd04a8244afb9fda5a0de"
server_key_iv: "5bd29de94c43328adbfcf6581554aa73"
server_key_version: "encryptionkey@0"
I20260927 16:32:14.765377 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:14.767649 3514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:14.768787 3499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:32:14.768920 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "6b60c196b383486db4c5d13141d58e9a"
format_stamp: "Formatted at 2026-09-27 16:32:14 on dist-test-slave-grzf"
server_key: "073826a877ffd04a8244afb9fda5a0de"
server_key_iv: "5bd29de94c43328adbfcf6581554aa73"
server_key_version: "encryptionkey@0"
I20260927 16:32:14.769023 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:14.783268 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:14.786321 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:14.786513 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:14.792966 3499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38721
I20260927 16:32:14.792975 3566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38721 every 8 connection(s)
I20260927 16:32:14.793848 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:14.796484 3567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:14.801671 3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a: Bootstrap starting.
I20260927 16:32:14.802443 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3499
I20260927 16:32:14.802597 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:14.802860 15734 external_mini_cluster.cc:1428] Setting key 2d120c825dd5fa60a86e8593d78f8af4
I20260927 16:32:14.804427 3567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:14.805266 3567 log.cc:824] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:14.807307 3567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a: No bootstrap required, opened a new log
I20260927 16:32:14.811162 3567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260927 16:32:14.811421 3567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:14.811486 3567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60c196b383486db4c5d13141d58e9a, State: Initialized, Role: FOLLOWER
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:32:14.811967 3567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260927 16:32:14.812143 3567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:14.812219 3567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:14.812330 3567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:14.813189 3567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260927 16:32:14.813508 3567 leader_election.cc:302] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b60c196b383486db4c5d13141d58e9a; no voters:
I20260927 16:32:14.813827 3567 leader_election.cc:288] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:14.813884 3572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:14.814064 3572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [term 1 LEADER]: Becoming Leader. State: Replica: 6b60c196b383486db4c5d13141d58e9a, State: Running, Role: LEADER
I20260927 16:32:14.814394 3572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } }
I20260927 16:32:14.815038 3567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:14.815536 3572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b60c196b383486db4c5d13141d58e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260927 16:32:14.815656 3572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:14.816045 3573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b60c196b383486db4c5d13141d58e9a. Latest consensus state: current_term: 1 leader_uuid: "6b60c196b383486db4c5d13141d58e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b60c196b383486db4c5d13141d58e9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38721 } } }
I20260927 16:32:14.816049 3576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:14.816180 3573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:14.819209 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:14.803993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38477 (local address 127.0.0.1:38721)
0927 16:32:14.804502 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:14.804521 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:14.804563 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:32:14.805397 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:14.805419 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:14.805818 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:14.806164 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:14.806813 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:14.808251 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:14.809081 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:14.809800 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:14.809987 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:14.812617 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:14.812621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:14.812633 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:14.812658 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:14.814732 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:14.815285 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:14.815289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:14.815293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:14.815472 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:14.816613 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:14.816617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:14.816618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:14.817088 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:14.817892 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:14.818076 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 16:32:14.818291 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":118,"threads_started":1}
I20260927 16:32:14.819391 3576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:14.822050 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:14.825352 3576 catalog_manager.cc:1347] Generated new cluster ID: 1d386f8949a5495d89c4773149297ca4
I20260927 16:32:14.825416 3576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:14.832947 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:14.824397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59135 (local address 127.0.0.1:38721)
0927 16:32:14.824496 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:14.824501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:14.825662 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 16:32:14.825782 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:14.825785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:14.825826 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:14.825916 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:14.826552 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:14.827084 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:14.827953 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:14.828199 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:14.828312 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:14.829691 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:14.829694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:14.829696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:14.829721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:14.831163 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:14.831676 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:14.831679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:14.831681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:14.831758 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:14.832081 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:14.832085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:14.832086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:14.832254 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:14.832403 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:14.832670 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 16:32:14.832761 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:14.850625 3576 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:14.851722 3576 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:14.862349 3576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b60c196b383486db4c5d13141d58e9a: Generated new TSK 0
I20260927 16:32:14.863003 3576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:14.869520 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3499
W20260927 16:32:14.874776 3484 connection.cc:535] client connection to 127.0.0.1:38721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38721): Connection reset by peer (error 104)
2026-09-27T16:32:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3598](info): set up 2 sockets
Sep 27 16:32:14 dist-test-slave-grzf krb5kdc[3598](info): commencing operation
krb5kdc: starting...
W20260927 16:32:15.903537 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:15 dist-test-slave-grzf krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:16.024158 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:16.133402 3611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:16.133663 3611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:16.133705 3611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:16.137792 3611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:16.137849 3611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:16.137868 3611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:16.137885 3611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:16.137900 3611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:16.145689 3611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:16.147408 3611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:16.148981 3611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:16.157513 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:16.157509 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:16.157550 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:16.157904 3611 server_base.cc:1034] running on GCE node
I20260927 16:32:16.158485 3611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:16.159807 3611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:16.160971 3611 hybrid_clock.cc:648] HybridClock initialized: now 1790526736160948 us; error 40 us; skew 500 ppm
Sep 27 16:32:16 dist-test-slave-grzf krb5kdc[3598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:16.163691 3611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:16.164525 3611 webserver.cc:460] Webserver started at http://127.0.0.1:34041/ using document root <none> and password file <none>
I20260927 16:32:16.165043 3611 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:16.165098 3611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:16.165264 3611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:16.167237 3611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4"
format_stamp: "Formatted at 2026-09-27 16:32:16 on dist-test-slave-grzf"
server_key: "298486762efc84ffe378119982dc16d5"
server_key_iv: "bc5b4b896d2b66ff985b9c33436c8547"
server_key_version: "encryptionkey@0"
I20260927 16:32:16.167831 3611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4"
format_stamp: "Formatted at 2026-09-27 16:32:16 on dist-test-slave-grzf"
server_key: "298486762efc84ffe378119982dc16d5"
server_key_iv: "bc5b4b896d2b66ff985b9c33436c8547"
server_key_version: "encryptionkey@0"
I20260927 16:32:16.171362 3611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:32:16.173708 3626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:16.174896 3611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:32:16.175025 3611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4"
format_stamp: "Formatted at 2026-09-27 16:32:16 on dist-test-slave-grzf"
server_key: "298486762efc84ffe378119982dc16d5"
server_key_iv: "bc5b4b896d2b66ff985b9c33436c8547"
server_key_version: "encryptionkey@0"
I20260927 16:32:16.175127 3611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:16.209892 3611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:16.212980 3611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:16.213191 3611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:16.220118 3611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46823
I20260927 16:32:16.220152 3678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46823 every 8 connection(s)
I20260927 16:32:16.221063 3611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:16.223812 3679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:16.229115 3679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4: Bootstrap starting.
I20260927 16:32:16.230463 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3611
I20260927 16:32:16.230615 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:16.230921 15734 external_mini_cluster.cc:1428] Setting key 03aeac5c04d6aed5c9523bb3a8f63cff
I20260927 16:32:16.231947 3679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:16.232749 3679 log.cc:824] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:16.234843 3679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4: No bootstrap required, opened a new log
I20260927 16:32:16.238763 3679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260927 16:32:16.239044 3679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:16.239125 3679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfa3e2a7ea6497db4ce2130984a1cc4, State: Initialized, Role: FOLLOWER
I20260927 16:32:16.239646 3679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260927 16:32:16.239770 3679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:16.239856 3679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:16.239964 3679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:16.240042 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:16.232048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49055 (local address 127.0.0.1:46823)
0927 16:32:16.232511 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:16.232527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:16.232560 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:16.233355 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:16.233381 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:16.233756 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:16.234089 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:16.234509 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:16.235564 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:16.236433 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:16.236915 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:16.237096 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:16.238050 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:16.238056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:16.238069 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:16.238451 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:16.238512 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:16.238535 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:16.238694 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:16.238896 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 16:32:16.239062 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260927 16:32:16.240944 3679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260927 16:32:16.241308 3679 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfa3e2a7ea6497db4ce2130984a1cc4; no voters:
I20260927 16:32:16.241653 3679 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:16.241751 3684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:16.241976 3684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [term 1 LEADER]: Becoming Leader. State: Replica: 9dfa3e2a7ea6497db4ce2130984a1cc4, State: Running, Role: LEADER
I20260927 16:32:16.242302 3684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } }
I20260927 16:32:16.243011 3679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:16.243281 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:16.243956 3685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } } }
I20260927 16:32:16.244081 3685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:16.244302 3685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfa3e2a7ea6497db4ce2130984a1cc4. Latest consensus state: current_term: 1 leader_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfa3e2a7ea6497db4ce2130984a1cc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46823 } } }
I20260927 16:32:16.244391 3685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:16.244998 3693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:16.248132 3693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:16.249877 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:16.245923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41665 (local address 127.0.0.1:46823)
0927 16:32:16.246068 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:16.246072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:16.246219 (+ 147us) server_negotiation.cc:407] Connection header received
0927 16:32:16.246336 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:16.246339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:16.246387 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:16.246500 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:16.246766 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:16.247287 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:16.248134 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:16.248433 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:16.248544 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:16.249421 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:16.249424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:16.249426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:16.249438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:16.249468 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:16.249479 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:16.249572 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:16.249693 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 16:32:16.249727 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 16:32:16.253808 3693 catalog_manager.cc:1347] Generated new cluster ID: 304afa8092d342b789a6ae2287943f90
I20260927 16:32:16.253876 3693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:16.284446 3693 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:16.285468 3693 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:16.290498 3693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfa3e2a7ea6497db4ce2130984a1cc4: Generated new TSK 0
I20260927 16:32:16.291487 3693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:16.304116 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3611
W20260927 16:32:16.308853 3596 connection.cc:535] client connection to 127.0.0.1:46823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46823): Connection reset by peer (error 104)
2026-09-27T16:32:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:16 dist-test-slave-grzf krb5kdc[3710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:16 dist-test-slave-grzf krb5kdc[3710](info): set up 2 sockets
Sep 27 16:32:16 dist-test-slave-grzf krb5kdc[3710](info): commencing operation
krb5kdc: starting...
W20260927 16:32:17.403434 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:17 dist-test-slave-grzf krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:17.520460 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:17.627319 3723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:17.627647 3723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:17.627727 3723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:17.631651 3723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:17.631721 3723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:17.631755 3723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:17.631810 3723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:17.631847 3723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:17.636912 3723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:17.637863 3723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:17.638825 3723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:17.647002 3728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:17.647001 3729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:17.647007 3731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:17.647253 3723 server_base.cc:1034] running on GCE node
I20260927 16:32:17.647842 3723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:17.648797 3723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:17.649950 3723 hybrid_clock.cc:648] HybridClock initialized: now 1790526737649932 us; error 29 us; skew 500 ppm
Sep 27 16:32:17 dist-test-slave-grzf krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:17.652521 3723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:17.653316 3723 webserver.cc:460] Webserver started at http://127.0.0.1:46375/ using document root <none> and password file <none>
I20260927 16:32:17.653802 3723 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:17.653859 3723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:17.654007 3723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:17.655937 3723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "d70e46677b0e439fb67eb410929a0832"
format_stamp: "Formatted at 2026-09-27 16:32:17 on dist-test-slave-grzf"
server_key: "22fa2341f29c6f944d2c31a116be11f8"
server_key_iv: "42c74c40b1672baf8436943555533192"
server_key_version: "encryptionkey@0"
I20260927 16:32:17.656418 3723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "d70e46677b0e439fb67eb410929a0832"
format_stamp: "Formatted at 2026-09-27 16:32:17 on dist-test-slave-grzf"
server_key: "22fa2341f29c6f944d2c31a116be11f8"
server_key_iv: "42c74c40b1672baf8436943555533192"
server_key_version: "encryptionkey@0"
I20260927 16:32:17.659813 3723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:17.661959 3738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:17.663122 3723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:17.663218 3723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "d70e46677b0e439fb67eb410929a0832"
format_stamp: "Formatted at 2026-09-27 16:32:17 on dist-test-slave-grzf"
server_key: "22fa2341f29c6f944d2c31a116be11f8"
server_key_iv: "42c74c40b1672baf8436943555533192"
server_key_version: "encryptionkey@0"
I20260927 16:32:17.663297 3723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:17.667601 3723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:17.670310 3723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:17.670442 3723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:17.677145 3723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45527
I20260927 16:32:17.677171 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45527 every 8 connection(s)
I20260927 16:32:17.678001 3723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:17.680576 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:17.686017 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832: Bootstrap starting.
I20260927 16:32:17.686174 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3723
I20260927 16:32:17.686317 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:17.686619 15734 external_mini_cluster.cc:1428] Setting key 08d0096bd8b645be67061b8b3c943bd2
I20260927 16:32:17.688657 3791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:17.689364 3791 log.cc:824] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:17.691574 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832: No bootstrap required, opened a new log
I20260927 16:32:17.691936 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:17.687926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51733 (local address 127.0.0.1:45527)
0927 16:32:17.688360 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:17.688373 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:17.688406 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:17.689063 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:17.689082 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:17.689357 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:17.689716 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:17.690505 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:17.690511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:17.690526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:17.690537 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:17.690608 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:17.690635 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:17.690741 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:17.690869 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 16:32:17.691015 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
I20260927 16:32:17.694586 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:17.695613 3791 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260927 16:32:17.695842 3791 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:17.695924 3791 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70e46677b0e439fb67eb410929a0832, State: Initialized, Role: FOLLOWER
I20260927 16:32:17.696372 3791 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260927 16:32:17.696514 3791 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:17.696611 3791 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:17.696722 3791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:17.697713 3791 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260927 16:32:17.698104 3791 leader_election.cc:302] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70e46677b0e439fb67eb410929a0832; no voters:
I20260927 16:32:17.698243 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:17.696648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56451 (local address 127.0.0.1:45527)
0927 16:32:17.696758 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:17.696762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:17.696943 (+ 181us) server_negotiation.cc:407] Connection header received
0927 16:32:17.697020 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:17.697025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:17.697062 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:17.697153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:17.697809 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:17.697814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:17.697817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:17.697828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:17.697853 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:17.697864 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:17.697949 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:17.698060 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 16:32:17.698095 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:32:17.698448 3791 leader_election.cc:288] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:17.698827 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260927 16:32:17.698968 3801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:17.698995 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:17.699240 3801 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [term 1 LEADER]: Becoming Leader. State: Replica: d70e46677b0e439fb67eb410929a0832, State: Running, Role: LEADER
I20260927 16:32:17.699544 3801 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260927 16:32:17.699676 3791 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:17.700894 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70e46677b0e439fb67eb410929a0832. Latest consensus state: current_term: 1 leader_uuid: "d70e46677b0e439fb67eb410929a0832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70e46677b0e439fb67eb410929a0832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260927 16:32:17.700999 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:17.701514 3812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:17.704668 3812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:17.710012 3812 catalog_manager.cc:1347] Generated new cluster ID: 9903d6a2e59547f597f4934a8ebb7258
I20260927 16:32:17.710088 3812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:17.742679 3812 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:17.743062 3812 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:17.757791 3812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70e46677b0e439fb67eb410929a0832: Generated new TSK 0
I20260927 16:32:17.758533 3812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:17.769583 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3723
2026-09-27T16:32:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:17 dist-test-slave-grzf krb5kdc[3822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:17 dist-test-slave-grzf krb5kdc[3822](info): set up 2 sockets
Sep 27 16:32:17 dist-test-slave-grzf krb5kdc[3822](info): commencing operation
krb5kdc: starting...
W20260927 16:32:18.791560 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:18 dist-test-slave-grzf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:18.906250 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:19.013720 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:19.013974 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:19.014014 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:19.017858 3835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:19.017910 3835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:19.017928 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:19.017944 3835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:19.017959 3835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:19.022895 3835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:19.023830 3835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:19.024740 3835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:19.032660 3840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:19.032646 3843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:19.032698 3841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:19.033032 3835 server_base.cc:1034] running on GCE node
I20260927 16:32:19.033524 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:19.034511 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:19.035665 3835 hybrid_clock.cc:648] HybridClock initialized: now 1790526739035646 us; error 28 us; skew 500 ppm
Sep 27 16:32:19 dist-test-slave-grzf krb5kdc[3822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:19.041363 3835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:19.042140 3835 webserver.cc:460] Webserver started at http://127.0.0.1:40309/ using document root <none> and password file <none>
I20260927 16:32:19.042630 3835 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:19.042685 3835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:19.042835 3835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:19.044889 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "d282db1fa49241d4ae1393a5fef2704a"
format_stamp: "Formatted at 2026-09-27 16:32:19 on dist-test-slave-grzf"
server_key: "2189a9e4f3d6011c747a7ff553903b34"
server_key_iv: "7d4860ebdf37a9940553ea676a42a751"
server_key_version: "encryptionkey@0"
I20260927 16:32:19.045377 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "d282db1fa49241d4ae1393a5fef2704a"
format_stamp: "Formatted at 2026-09-27 16:32:19 on dist-test-slave-grzf"
server_key: "2189a9e4f3d6011c747a7ff553903b34"
server_key_iv: "7d4860ebdf37a9940553ea676a42a751"
server_key_version: "encryptionkey@0"
I20260927 16:32:19.048803 3835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:19.051031 3850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:19.052301 3835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:19.052418 3835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "d282db1fa49241d4ae1393a5fef2704a"
format_stamp: "Formatted at 2026-09-27 16:32:19 on dist-test-slave-grzf"
server_key: "2189a9e4f3d6011c747a7ff553903b34"
server_key_iv: "7d4860ebdf37a9940553ea676a42a751"
server_key_version: "encryptionkey@0"
I20260927 16:32:19.052520 3835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:19.071724 3835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:19.074781 3835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:19.074990 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:19.081939 3835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34599
I20260927 16:32:19.081965 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34599 every 8 connection(s)
I20260927 16:32:19.082850 3835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:19.085557 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:19.091037 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: Bootstrap starting.
I20260927 16:32:19.092096 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3835
I20260927 16:32:19.092240 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:19.092530 15734 external_mini_cluster.cc:1428] Setting key 0ba383ced9fc2b365e5055df79ba111e
I20260927 16:32:19.093850 3903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:19.094756 3903 log.cc:824] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:19.096872 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: No bootstrap required, opened a new log
I20260927 16:32:19.100971 3903 raft_consensus.cc:348] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } }
I20260927 16:32:19.101174 3903 raft_consensus.cc:374] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:19.101253 3903 raft_consensus.cc:729] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d282db1fa49241d4ae1393a5fef2704a, State: Initialized, Role: FOLLOWER
Sep 27 16:32:19 dist-test-slave-grzf krb5kdc[3822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:32:19.101754 3903 consensus_queue.cc:260] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } }
I20260927 16:32:19.101891 3903 raft_consensus.cc:388] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:19.101975 3903 raft_consensus.cc:482] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:19.102067 3903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:19.103158 3903 raft_consensus.cc:504] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } }
I20260927 16:32:19.103574 3903 leader_election.cc:302] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d282db1fa49241d4ae1393a5fef2704a; no voters:
I20260927 16:32:19.103935 3903 leader_election.cc:288] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:19.104009 3908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:19.104187 3908 raft_consensus.cc:686] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [term 1 LEADER]: Becoming Leader. State: Replica: d282db1fa49241d4ae1393a5fef2704a, State: Running, Role: LEADER
I20260927 16:32:19.104526 3908 consensus_queue.cc:237] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } }
I20260927 16:32:19.105191 3903 sys_catalog.cc:564] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:19.106232 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d282db1fa49241d4ae1393a5fef2704a. Latest consensus state: current_term: 1 leader_uuid: "d282db1fa49241d4ae1393a5fef2704a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } } }
I20260927 16:32:19.106361 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:19.106726 3909 sys_catalog.cc:455] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d282db1fa49241d4ae1393a5fef2704a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d282db1fa49241d4ae1393a5fef2704a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34599 } } }
I20260927 16:32:19.107359 3909 sys_catalog.cc:458] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:19.108137 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:19.093619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60463 (local address 127.0.0.1:34599)
0927 16:32:19.094116 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:19.094137 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:19.094181 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:32:19.094985 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:19.095009 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:19.095447 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:19.095813 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:19.096480 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:19.097959 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:19.098792 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:19.099540 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:19.099707 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:19.102298 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:19.102302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:19.102314 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:19.102339 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:19.104219 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:19.104717 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:19.104722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:19.104728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:19.104842 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:19.105131 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:19.105135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:19.105137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:19.105661 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:19.106533 (+ 872us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:19.106678 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 16:32:19.106968 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":135,"threads_started":1}
W20260927 16:32:19.110450 3924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:19.110526 3924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:19.110584 3917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:19.110769 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:19.111447 3917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:19.116914 3917 catalog_manager.cc:1347] Generated new cluster ID: 1a4472b1dabb4f85a7ac845bdcfa8f89
I20260927 16:32:19.116974 3917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:19.124035 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:19.112779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60283 (local address 127.0.0.1:34599)
0927 16:32:19.112883 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:19.112888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:19.113492 (+ 604us) server_negotiation.cc:407] Connection header received
0927 16:32:19.113592 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:19.113595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:19.113639 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:19.113724 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:19.114515 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:19.115656 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:19.116847 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:19.117163 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:19.117232 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:19.120890 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:19.120894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:19.120897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:19.120926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:19.122394 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:19.122812 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:19.122816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:19.122818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:19.122896 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:19.123168 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:19.123172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:19.123174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:19.123344 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:19.123530 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:19.123756 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 16:32:19.123862 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 16:32:19.131060 3917 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:19.132267 3917 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:19.149482 3917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d282db1fa49241d4ae1393a5fef2704a: Generated new TSK 0
I20260927 16:32:19.150333 3917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:19.157547 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3835
2026-09-27T16:32:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:19 dist-test-slave-grzf krb5kdc[3934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:19 dist-test-slave-grzf krb5kdc[3934](info): set up 2 sockets
Sep 27 16:32:19 dist-test-slave-grzf krb5kdc[3934](info): commencing operation
krb5kdc: starting...
W20260927 16:32:20.195508 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:20 dist-test-slave-grzf krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:20.314083 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:20.421887 3947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:20.422143 3947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:20.422184 3947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:20.426226 3947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:20.426286 3947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:20.426306 3947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:20.426322 3947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:20.426337 3947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:20.431411 3947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:20.432348 3947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:20.433259 3947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:20.441484 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:20.441598 3947 server_base.cc:1034] running on GCE node
W20260927 16:32:20.441479 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:20.441479 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:20.442271 3947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:20.443282 3947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:20.444468 3947 hybrid_clock.cc:648] HybridClock initialized: now 1790526740444449 us; error 37 us; skew 500 ppm
Sep 27 16:32:20 dist-test-slave-grzf krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:20.446964 3947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:20.447971 3947 webserver.cc:460] Webserver started at http://127.0.0.1:37023/ using document root <none> and password file <none>
I20260927 16:32:20.448743 3947 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:20.448873 3947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:20.449178 3947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:20.451352 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "e7cbf912fc2b4edf90a1efce03981aee"
format_stamp: "Formatted at 2026-09-27 16:32:20 on dist-test-slave-grzf"
server_key: "0d0dab0b5be51b8372bc9719d494652a"
server_key_iv: "cdd1826eff132830ba0d53d6df211f84"
server_key_version: "encryptionkey@0"
I20260927 16:32:20.451947 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "e7cbf912fc2b4edf90a1efce03981aee"
format_stamp: "Formatted at 2026-09-27 16:32:20 on dist-test-slave-grzf"
server_key: "0d0dab0b5be51b8372bc9719d494652a"
server_key_iv: "cdd1826eff132830ba0d53d6df211f84"
server_key_version: "encryptionkey@0"
I20260927 16:32:20.455504 3947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:20.457703 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:20.458844 3947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:20.458981 3947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "e7cbf912fc2b4edf90a1efce03981aee"
format_stamp: "Formatted at 2026-09-27 16:32:20 on dist-test-slave-grzf"
server_key: "0d0dab0b5be51b8372bc9719d494652a"
server_key_iv: "cdd1826eff132830ba0d53d6df211f84"
server_key_version: "encryptionkey@0"
I20260927 16:32:20.459107 3947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:20.469338 3947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:20.472029 3947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:20.472219 3947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:20.479084 3947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42827
I20260927 16:32:20.479135 4014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42827 every 8 connection(s)
I20260927 16:32:20.480055 3947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:20.482672 4015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:20.487824 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee: Bootstrap starting.
I20260927 16:32:20.489771 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 3947
I20260927 16:32:20.489933 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:20.490199 15734 external_mini_cluster.cc:1428] Setting key 2727812171cf31a95896bd33febe4f00
I20260927 16:32:20.490211 4015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:20.491071 4015 log.cc:824] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:20.492960 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee: No bootstrap required, opened a new log
I20260927 16:32:20.497005 4015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } }
I20260927 16:32:20.497213 4015 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:20.497277 4015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cbf912fc2b4edf90a1efce03981aee, State: Initialized, Role: FOLLOWER
I20260927 16:32:20.497766 4015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } }
I20260927 16:32:20.497944 4015 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:20.498010 4015 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:20.498112 4015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:20.498708 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:20.491287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44605 (local address 127.0.0.1:42827)
0927 16:32:20.491745 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:20.491760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:20.491794 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:32:20.492391 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:20.492406 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:20.492716 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:20.492981 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:20.493348 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:20.494239 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:20.495503 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:20.495959 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:20.496082 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:20.496883 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:20.496888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:20.496902 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:20.497336 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:20.497407 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:20.497430 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:20.497502 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:20.497673 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 16:32:20.497887 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260927 16:32:20.498979 4015 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } }
I20260927 16:32:20.499320 4015 leader_election.cc:302] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7cbf912fc2b4edf90a1efce03981aee; no voters:
I20260927 16:32:20.499670 4015 leader_election.cc:288] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:20.499754 4020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:20.499943 4020 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [term 1 LEADER]: Becoming Leader. State: Replica: e7cbf912fc2b4edf90a1efce03981aee, State: Running, Role: LEADER
I20260927 16:32:20.500283 4020 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } }
I20260927 16:32:20.500911 4015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:20.501497 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:20.501863 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7cbf912fc2b4edf90a1efce03981aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } } }
I20260927 16:32:20.501962 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:20.502285 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7cbf912fc2b4edf90a1efce03981aee. Latest consensus state: current_term: 1 leader_uuid: "e7cbf912fc2b4edf90a1efce03981aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbf912fc2b4edf90a1efce03981aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42827 } } }
I20260927 16:32:20.502362 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:20.502611 4029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:20.506560 4029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:20.508443 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:20.503939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47443 (local address 127.0.0.1:42827)
0927 16:32:20.504032 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:20.504036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:20.504221 (+ 185us) server_negotiation.cc:407] Connection header received
0927 16:32:20.504303 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:20.504306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:20.504346 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:20.504424 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:20.504656 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:20.505364 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:20.506460 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:20.506771 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:20.507227 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:20.507778 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:20.507783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:20.507786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:20.507797 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:20.507823 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:20.507836 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:20.508243 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:20.508276 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 16:32:20.508325 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 16:32:20.511889 4029 catalog_manager.cc:1347] Generated new cluster ID: f923dffa7ceb40d8a0edee37a0030126
I20260927 16:32:20.511947 4029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:20.525388 4029 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:20.526602 4029 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:20.533599 4029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7cbf912fc2b4edf90a1efce03981aee: Generated new TSK 0
I20260927 16:32:20.534395 4029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:20.543610 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 3947
2026-09-27T16:32:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:20 dist-test-slave-grzf krb5kdc[4046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:20 dist-test-slave-grzf krb5kdc[4046](info): set up 2 sockets
Sep 27 16:32:20 dist-test-slave-grzf krb5kdc[4046](info): commencing operation
krb5kdc: starting...
W20260927 16:32:21.567654 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:21 dist-test-slave-grzf krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:21.684473 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:21.791706 4059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:21.791976 4059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:21.792055 4059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:21.796141 4059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:21.796216 4059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:21.796254 4059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:21.796310 4059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:21.796348 4059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:21.801383 4059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:21.802366 4059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:21.803316 4059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:21.811240 4065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:21.811255 4067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:21.811251 4064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:21.811655 4059 server_base.cc:1034] running on GCE node
I20260927 16:32:21.812202 4059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:21.813270 4059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:21.814459 4059 hybrid_clock.cc:648] HybridClock initialized: now 1790526741814426 us; error 46 us; skew 500 ppm
Sep 27 16:32:21 dist-test-slave-grzf krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:21.817116 4059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:21.817917 4059 webserver.cc:460] Webserver started at http://127.0.0.1:46225/ using document root <none> and password file <none>
I20260927 16:32:21.818457 4059 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:21.818545 4059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:21.818820 4059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:21.821439 4059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "0d6441c6dc324646a6d90dbfe7ccf37d"
format_stamp: "Formatted at 2026-09-27 16:32:21 on dist-test-slave-grzf"
server_key: "f325d2b7e14894c63c19230713355203"
server_key_iv: "6db39bb0b8938fd9e81c0a44fa47b18b"
server_key_version: "encryptionkey@0"
I20260927 16:32:21.822039 4059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "0d6441c6dc324646a6d90dbfe7ccf37d"
format_stamp: "Formatted at 2026-09-27 16:32:21 on dist-test-slave-grzf"
server_key: "f325d2b7e14894c63c19230713355203"
server_key_iv: "6db39bb0b8938fd9e81c0a44fa47b18b"
server_key_version: "encryptionkey@0"
I20260927 16:32:21.825702 4059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:32:21.828213 4074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:21.829430 4059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:32:21.829593 4059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "0d6441c6dc324646a6d90dbfe7ccf37d"
format_stamp: "Formatted at 2026-09-27 16:32:21 on dist-test-slave-grzf"
server_key: "f325d2b7e14894c63c19230713355203"
server_key_iv: "6db39bb0b8938fd9e81c0a44fa47b18b"
server_key_version: "encryptionkey@0"
I20260927 16:32:21.829710 4059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:21.834205 4059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:21.837423 4059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:21.837617 4059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:21.844416 4059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42589
I20260927 16:32:21.844467 4126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42589 every 8 connection(s)
I20260927 16:32:21.845355 4059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:21.848012 4127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:21.850126 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4059
I20260927 16:32:21.850299 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:21.850569 15734 external_mini_cluster.cc:1428] Setting key d90ff89dcb62beec1633092d391f7829
I20260927 16:32:21.853782 4127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d: Bootstrap starting.
I20260927 16:32:21.855659 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:21.852035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34969 (local address 127.0.0.1:42589)
0927 16:32:21.852482 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:21.852496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:21.852530 (+ 34us) server_negotiation.cc:407] Connection header received
0927 16:32:21.853144 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:21.853159 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:21.853430 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:21.853761 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:21.854462 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:21.854466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:21.854478 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:21.854486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:21.854531 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:21.854549 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:21.854652 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:21.854749 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 16:32:21.854908 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260927 16:32:21.856254 4127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:21.857093 4127 log.cc:824] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:21.858383 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:21.859292 4127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d: No bootstrap required, opened a new log
I20260927 16:32:21.862087 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:21.860524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46695 (local address 127.0.0.1:42589)
0927 16:32:21.860648 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:21.860652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:21.860818 (+ 166us) server_negotiation.cc:407] Connection header received
0927 16:32:21.860917 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:21.860919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:21.860945 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:21.861009 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:21.861705 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:21.861708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:21.861710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:21.861720 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:21.861744 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:21.861755 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:21.861824 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:21.861920 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 16:32:21.861947 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 16:32:21.863865 4127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } }
I20260927 16:32:21.864100 4127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:21.864171 4127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6441c6dc324646a6d90dbfe7ccf37d, State: Initialized, Role: FOLLOWER
I20260927 16:32:21.864615 4127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } }
I20260927 16:32:21.864775 4127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:21.864847 4127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:21.864933 4127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:21.865713 4127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } }
I20260927 16:32:21.865990 4127 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d6441c6dc324646a6d90dbfe7ccf37d; no voters:
I20260927 16:32:21.866228 4127 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:21.866335 4137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:21.866561 4137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [term 1 LEADER]: Becoming Leader. State: Replica: 0d6441c6dc324646a6d90dbfe7ccf37d, State: Running, Role: LEADER
I20260927 16:32:21.866874 4137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } }
I20260927 16:32:21.867422 4127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:21.868624 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d6441c6dc324646a6d90dbfe7ccf37d. Latest consensus state: current_term: 1 leader_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } } }
I20260927 16:32:21.868621 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6441c6dc324646a6d90dbfe7ccf37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42589 } } }
I20260927 16:32:21.868773 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:21.868773 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:21.869182 4146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:21.872263 4146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:21.877753 4146 catalog_manager.cc:1347] Generated new cluster ID: afad2b44f3364093846131d6fa29c522
I20260927 16:32:21.877882 4146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:21.899274 4146 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:21.899646 4146 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:21.908087 4146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d6441c6dc324646a6d90dbfe7ccf37d: Generated new TSK 0
I20260927 16:32:21.908854 4146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:21.920639 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4059
2026-09-27T16:32:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:21 dist-test-slave-grzf krb5kdc[4158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:21 dist-test-slave-grzf krb5kdc[4158](info): set up 2 sockets
Sep 27 16:32:21 dist-test-slave-grzf krb5kdc[4158](info): commencing operation
krb5kdc: starting...
W20260927 16:32:22.967797 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:23.094714 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:23.203658 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:23.203934 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:23.204043 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:23.207978 4171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:23.208048 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:23.208115 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:23.208169 4171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:23.208206 4171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:23.213447 4171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:23.214437 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:23.215443 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:23.223651 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:23.223683 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:23.223703 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:23.223837 4171 server_base.cc:1034] running on GCE node
I20260927 16:32:23.224489 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:23.225534 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:23.226722 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790526743226692 us; error 40 us; skew 500 ppm
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:23.229400 4171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:23.230295 4171 webserver.cc:460] Webserver started at http://127.0.0.1:38485/ using document root <none> and password file <none>
I20260927 16:32:23.230858 4171 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:23.230950 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:23.231179 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:23.233203 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "ce1ae467629d4d398becd896b251a5be"
format_stamp: "Formatted at 2026-09-27 16:32:23 on dist-test-slave-grzf"
server_key: "f26ec7db531027dc02622fba19151db1"
server_key_iv: "d4e44987ec998c975d545f14db7ff124"
server_key_version: "encryptionkey@0"
I20260927 16:32:23.233762 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "ce1ae467629d4d398becd896b251a5be"
format_stamp: "Formatted at 2026-09-27 16:32:23 on dist-test-slave-grzf"
server_key: "f26ec7db531027dc02622fba19151db1"
server_key_iv: "d4e44987ec998c975d545f14db7ff124"
server_key_version: "encryptionkey@0"
I20260927 16:32:23.237319 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:23.239631 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:23.240873 4171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:23.241014 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "ce1ae467629d4d398becd896b251a5be"
format_stamp: "Formatted at 2026-09-27 16:32:23 on dist-test-slave-grzf"
server_key: "f26ec7db531027dc02622fba19151db1"
server_key_iv: "d4e44987ec998c975d545f14db7ff124"
server_key_version: "encryptionkey@0"
I20260927 16:32:23.241127 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:23.264599 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:23.267611 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:23.267822 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:23.274771 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40373
I20260927 16:32:23.274798 4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40373 every 8 connection(s)
I20260927 16:32:23.275722 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:23.278342 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:23.280589 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4171
I20260927 16:32:23.280741 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:23.281042 15734 external_mini_cluster.cc:1428] Setting key d844edf1793a0df628480590333f379b
I20260927 16:32:23.284337 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be: Bootstrap starting.
I20260927 16:32:23.286402 4239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:23.287106 4239 log.cc:824] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:23.288976 4239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be: No bootstrap required, opened a new log
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:32:23.292160 4239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } }
I20260927 16:32:23.292389 4239 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:23.292464 4239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1ae467629d4d398becd896b251a5be, State: Initialized, Role: FOLLOWER
I20260927 16:32:23.292960 4239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } }
I20260927 16:32:23.293099 4239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:23.293160 4239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:23.293283 4239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:23.294112 4239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } }
I20260927 16:32:23.294495 4239 leader_election.cc:302] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce1ae467629d4d398becd896b251a5be; no voters:
I20260927 16:32:23.294809 4239 leader_election.cc:288] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:23.294904 4244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:23.295113 4244 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [term 1 LEADER]: Becoming Leader. State: Replica: ce1ae467629d4d398becd896b251a5be, State: Running, Role: LEADER
I20260927 16:32:23.295423 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:23.282431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:56393 (local address 127.0.0.1:40373)
0927 16:32:23.282843 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:23.282858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:23.282891 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:23.283547 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:23.283565 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:23.283904 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:23.284232 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:23.285035 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:23.286008 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:23.286932 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:23.287494 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:23.287665 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:23.290060 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:23.290071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:23.290086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:23.290112 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:23.292250 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:23.292801 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:23.292811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:23.292815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:23.292920 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:23.293274 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:23.293277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:23.293279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:23.293698 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:23.293914 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:23.294241 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 16:32:23.294461 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260927 16:32:23.295491 4244 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } }
I20260927 16:32:23.296066 4239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:23.296666 4246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1ae467629d4d398becd896b251a5be. Latest consensus state: current_term: 1 leader_uuid: "ce1ae467629d4d398becd896b251a5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } } }
I20260927 16:32:23.296778 4246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:23.297446 4245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce1ae467629d4d398becd896b251a5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1ae467629d4d398becd896b251a5be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40373 } } }
I20260927 16:32:23.297590 4245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:23.298524 4250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:23.299144 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:23.301812 4250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:23.307888 4250 catalog_manager.cc:1347] Generated new cluster ID: 1cbdbe735b4e4e92b5bf54be91ee8e4b
I20260927 16:32:23.307946 4250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:23.310873 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:23.302330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:32917 (local address 127.0.0.1:40373)
0927 16:32:23.302443 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:23.302447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:23.302755 (+ 308us) server_negotiation.cc:407] Connection header received
0927 16:32:23.302901 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:23.302904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:23.302949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:23.303037 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:23.303816 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:23.304617 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:23.305642 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:23.305965 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:23.306080 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:23.307534 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:23.307538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:23.307542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:23.307570 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:23.309106 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:23.309591 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:23.309594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:23.309596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:23.309680 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:23.310007 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:23.310010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:23.310012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:23.310189 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:23.310361 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:23.310615 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 16:32:23.310705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:32:23.344218 4250 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:23.345590 4250 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:23.351605 4250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce1ae467629d4d398becd896b251a5be: Generated new TSK 0
I20260927 16:32:23.352422 4250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:23.363003 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4171
W20260927 16:32:23.368309 4156 connection.cc:535] client connection to 127.0.0.1:40373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40373): Connection reset by peer (error 104)
2026-09-27T16:32:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4270](info): set up 2 sockets
Sep 27 16:32:23 dist-test-slave-grzf krb5kdc[4270](info): commencing operation
krb5kdc: starting...
W20260927 16:32:24.391629 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:24 dist-test-slave-grzf krb5kdc[4270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:24.510200 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:24.616377 4283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:24.616644 4283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:24.616731 4283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:24.620672 4283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:24.620745 4283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:24.620791 4283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:24.620824 4283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:24.620878 4283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:24.625936 4283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:24.626888 4283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:24.627880 4283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:24.635963 4291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:24.636096 4288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:24.635987 4289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:24.635977 4283 server_base.cc:1034] running on GCE node
I20260927 16:32:24.636689 4283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:24.637770 4283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:24.638929 4283 hybrid_clock.cc:648] HybridClock initialized: now 1790526744638916 us; error 25 us; skew 500 ppm
Sep 27 16:32:24 dist-test-slave-grzf krb5kdc[4270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:24.645584 4283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:24.646713 4283 webserver.cc:460] Webserver started at http://127.0.0.1:34257/ using document root <none> and password file <none>
I20260927 16:32:24.647539 4283 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:24.647646 4283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:24.647922 4283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:24.650017 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "d500079553994f86bf03efccb75340af"
format_stamp: "Formatted at 2026-09-27 16:32:24 on dist-test-slave-grzf"
server_key: "d1dab45b2a8e860d2bf679f2ee1dbe43"
server_key_iv: "4d824ec1adbfc55fdb19b1139c4fc924"
server_key_version: "encryptionkey@0"
I20260927 16:32:24.650568 4283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "d500079553994f86bf03efccb75340af"
format_stamp: "Formatted at 2026-09-27 16:32:24 on dist-test-slave-grzf"
server_key: "d1dab45b2a8e860d2bf679f2ee1dbe43"
server_key_iv: "4d824ec1adbfc55fdb19b1139c4fc924"
server_key_version: "encryptionkey@0"
I20260927 16:32:24.654084 4283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:24.656464 4298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:24.657624 4283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 16:32:24.657743 4283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "d500079553994f86bf03efccb75340af"
format_stamp: "Formatted at 2026-09-27 16:32:24 on dist-test-slave-grzf"
server_key: "d1dab45b2a8e860d2bf679f2ee1dbe43"
server_key_iv: "4d824ec1adbfc55fdb19b1139c4fc924"
server_key_version: "encryptionkey@0"
I20260927 16:32:24.657840 4283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:24.698468 4283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:24.701928 4283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:24.702142 4283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:24.709164 4283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36203
I20260927 16:32:24.709192 4350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36203 every 8 connection(s)
I20260927 16:32:24.710083 4283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:24.712702 4351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:24.715906 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4283
I20260927 16:32:24.716043 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:24.716346 15734 external_mini_cluster.cc:1428] Setting key fbf09e7100a4ac2701dc53d8c4379469
I20260927 16:32:24.718423 4351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af: Bootstrap starting.
I20260927 16:32:24.721021 4351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:24.721853 4351 log.cc:824] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:24.723889 4351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af: No bootstrap required, opened a new log
I20260927 16:32:24.726527 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:24.717673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:35321 (local address 127.0.0.1:36203)
0927 16:32:24.718074 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:24.718101 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:24.718142 (+ 41us) server_negotiation.cc:407] Connection header received
0927 16:32:24.718924 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:24.718944 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:24.719352 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:24.719756 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:24.720149 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:24.721617 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:24.722745 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:24.723438 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:24.723588 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:24.724391 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:24.724397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:24.724412 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:24.724867 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:24.724927 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:24.724954 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:24.725089 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:24.725223 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 16:32:24.725389 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260927 16:32:24.727639 4351 raft_consensus.cc:348] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260927 16:32:24.727838 4351 raft_consensus.cc:374] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:24.727896 4351 raft_consensus.cc:729] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d500079553994f86bf03efccb75340af, State: Initialized, Role: FOLLOWER
I20260927 16:32:24.728329 4351 consensus_queue.cc:260] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260927 16:32:24.728492 4351 raft_consensus.cc:388] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:24.728616 4351 raft_consensus.cc:482] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:24.728750 4351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:24.729519 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:24.729701 4351 raft_consensus.cc:504] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260927 16:32:24.730012 4351 leader_election.cc:302] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d500079553994f86bf03efccb75340af; no voters:
I20260927 16:32:24.730335 4351 leader_election.cc:288] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:24.730376 4356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:24.730545 4356 raft_consensus.cc:686] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [term 1 LEADER]: Becoming Leader. State: Replica: d500079553994f86bf03efccb75340af, State: Running, Role: LEADER
I20260927 16:32:24.730880 4356 consensus_queue.cc:237] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } }
I20260927 16:32:24.731539 4351 sys_catalog.cc:564] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:24.732035 4358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d500079553994f86bf03efccb75340af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260927 16:32:24.732221 4358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:24.732456 4357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [sys.catalog]: SysCatalogTable state changed. Reason: New leader d500079553994f86bf03efccb75340af. Latest consensus state: current_term: 1 leader_uuid: "d500079553994f86bf03efccb75340af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d500079553994f86bf03efccb75340af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36203 } } }
I20260927 16:32:24.732596 4357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:24.732703 4367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:24.735769 4367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:24.739950 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:24.733786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:44841 (local address 127.0.0.1:36203)
0927 16:32:24.733886 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:24.733892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:24.734120 (+ 228us) server_negotiation.cc:407] Connection header received
0927 16:32:24.734263 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:24.734266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:24.734313 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:24.734390 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:24.734700 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:24.735547 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:24.736612 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:24.736952 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:24.737065 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:24.739416 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:24.739420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:24.739422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:24.739435 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:24.739467 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:24.739479 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:24.739587 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:24.739774 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 16:32:24.739813 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:32:24.741176 4367 catalog_manager.cc:1347] Generated new cluster ID: 84311152285347ea9989717dbf365bd6
I20260927 16:32:24.741250 4367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:24.760675 4367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:24.761976 4367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:24.772264 4367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d500079553994f86bf03efccb75340af: Generated new TSK 0
I20260927 16:32:24.772915 4367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:24.780911 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4283
W20260927 16:32:24.786217 4268 connection.cc:535] client connection to 127.0.0.1:36203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36203): Connection reset by peer (error 104)
2026-09-27T16:32:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:24 dist-test-slave-grzf krb5kdc[4382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:24 dist-test-slave-grzf krb5kdc[4382](info): set up 2 sockets
Sep 27 16:32:24 dist-test-slave-grzf krb5kdc[4382](info): commencing operation
krb5kdc: starting...
W20260927 16:32:25.807663 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:25 dist-test-slave-grzf krb5kdc[4382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:25.924225 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:26.033464 4395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:26.033880 4395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:26.034019 4395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:26.038108 4395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:26.038201 4395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:26.038240 4395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:26.038300 4395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:26.038338 4395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:26.043560 4395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:26.044624 4395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:26.045630 4395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:26.054128 4400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:26.054131 4401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:26.054155 4403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:26.054714 4395 server_base.cc:1034] running on GCE node
I20260927 16:32:26.055445 4395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:26.056519 4395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:26.057713 4395 hybrid_clock.cc:648] HybridClock initialized: now 1790526746057700 us; error 35 us; skew 500 ppm
Sep 27 16:32:26 dist-test-slave-grzf krb5kdc[4382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:26.060596 4395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:26.061480 4395 webserver.cc:460] Webserver started at http://127.0.0.1:34053/ using document root <none> and password file <none>
I20260927 16:32:26.062000 4395 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:26.062055 4395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:26.062211 4395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:26.064194 4395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "fc09353b664a496094aa54e32aba54ab"
format_stamp: "Formatted at 2026-09-27 16:32:26 on dist-test-slave-grzf"
server_key: "c2a5fb2ed507c586fd3295eae351b4c9"
server_key_iv: "9bc6fcefb7ffdf7fc2a1535699eed8bb"
server_key_version: "encryptionkey@0"
I20260927 16:32:26.064680 4395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "fc09353b664a496094aa54e32aba54ab"
format_stamp: "Formatted at 2026-09-27 16:32:26 on dist-test-slave-grzf"
server_key: "c2a5fb2ed507c586fd3295eae351b4c9"
server_key_iv: "9bc6fcefb7ffdf7fc2a1535699eed8bb"
server_key_version: "encryptionkey@0"
I20260927 16:32:26.068257 4395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:26.070492 4410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:26.071794 4395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:26.071923 4395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "fc09353b664a496094aa54e32aba54ab"
format_stamp: "Formatted at 2026-09-27 16:32:26 on dist-test-slave-grzf"
server_key: "c2a5fb2ed507c586fd3295eae351b4c9"
server_key_iv: "9bc6fcefb7ffdf7fc2a1535699eed8bb"
server_key_version: "encryptionkey@0"
I20260927 16:32:26.072024 4395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:26.076562 4395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:26.079753 4395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:26.079941 4395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:26.086606 4395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44711
I20260927 16:32:26.086634 4462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44711 every 8 connection(s)
I20260927 16:32:26.087574 4395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:26.089833 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4395
I20260927 16:32:26.089967 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:26.090252 15734 external_mini_cluster.cc:1428] Setting key e88fd104ff2defacd718bfc0c97b9ee3
I20260927 16:32:26.090260 4463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:26.095266 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:26.091609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:38559 (local address 127.0.0.1:44711)
0927 16:32:26.092006 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:26.092019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:26.092052 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:26.092835 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:26.092852 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:26.093119 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:26.093440 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:26.094168 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:26.094173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:26.094184 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:26.094193 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:26.094240 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:26.094257 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:26.094400 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:26.094482 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 16:32:26.094640 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":131,"threads_started":1}
I20260927 16:32:26.096832 4463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab: Bootstrap starting.
I20260927 16:32:26.097954 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:26.099189 4463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:26.100082 4463 log.cc:824] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:26.101547 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:26.099940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:33809 (local address 127.0.0.1:44711)
0927 16:32:26.100046 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:26.100052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:26.100220 (+ 168us) server_negotiation.cc:407] Connection header received
0927 16:32:26.100338 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:26.100340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:26.100368 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:26.100423 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:26.101173 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:26.101176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:26.101178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:26.101195 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:26.101217 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:26.101227 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:26.101293 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:26.101392 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 16:32:26.101421 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:32:26.102167 4463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab: No bootstrap required, opened a new log
I20260927 16:32:26.105613 4463 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260927 16:32:26.105814 4463 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:26.105847 4463 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc09353b664a496094aa54e32aba54ab, State: Initialized, Role: FOLLOWER
I20260927 16:32:26.106184 4463 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260927 16:32:26.106288 4463 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:26.106330 4463 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:26.106396 4463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:26.107157 4463 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260927 16:32:26.107502 4463 leader_election.cc:302] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc09353b664a496094aa54e32aba54ab; no voters:
I20260927 16:32:26.107780 4463 leader_election.cc:288] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:26.107851 4473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:26.108040 4473 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [term 1 LEADER]: Becoming Leader. State: Replica: fc09353b664a496094aa54e32aba54ab, State: Running, Role: LEADER
I20260927 16:32:26.108378 4473 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } }
I20260927 16:32:26.109023 4463 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:26.110037 4474 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc09353b664a496094aa54e32aba54ab. Latest consensus state: current_term: 1 leader_uuid: "fc09353b664a496094aa54e32aba54ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260927 16:32:26.110176 4474 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:26.110462 4475 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc09353b664a496094aa54e32aba54ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc09353b664a496094aa54e32aba54ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44711 } } }
I20260927 16:32:26.110540 4475 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:26.110546 4482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:26.113725 4482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:26.119316 4482 catalog_manager.cc:1347] Generated new cluster ID: 126b036a9ff04a49b5c3b9cb1cd9a25e
I20260927 16:32:26.119405 4482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:26.135281 4482 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:26.135664 4482 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:26.144811 4482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc09353b664a496094aa54e32aba54ab: Generated new TSK 0
I20260927 16:32:26.146029 4482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:26.151171 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4395
2026-09-27T16:32:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:26 dist-test-slave-grzf krb5kdc[4494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:26 dist-test-slave-grzf krb5kdc[4494](info): set up 2 sockets
Sep 27 16:32:26 dist-test-slave-grzf krb5kdc[4494](info): commencing operation
krb5kdc: starting...
W20260927 16:32:27.183629 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:27.300612 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:27.412627 4507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:27.412923 4507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:27.412997 4507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:27.417042 4507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:27.417125 4507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:27.417164 4507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:27.417220 4507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:27.417259 4507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:27.422461 4507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:27.423575 4507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:27.424580 4507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:27.433172 4515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:27.433192 4512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:27.433183 4513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:27.433241 4507 server_base.cc:1034] running on GCE node
I20260927 16:32:27.434015 4507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:27.435036 4507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:27.436203 4507 hybrid_clock.cc:648] HybridClock initialized: now 1790526747436174 us; error 40 us; skew 500 ppm
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:27.438823 4507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:27.439769 4507 webserver.cc:460] Webserver started at http://127.0.0.1:42301/ using document root <none> and password file <none>
I20260927 16:32:27.440315 4507 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:27.440405 4507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:27.440634 4507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:27.442673 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc"
format_stamp: "Formatted at 2026-09-27 16:32:27 on dist-test-slave-grzf"
server_key: "e2a2a0a4134ba36a7b799ec2994c1be4"
server_key_iv: "36b70f49d9c5ff65e9f11d123e5d7ce9"
server_key_version: "encryptionkey@0"
I20260927 16:32:27.443240 4507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc"
format_stamp: "Formatted at 2026-09-27 16:32:27 on dist-test-slave-grzf"
server_key: "e2a2a0a4134ba36a7b799ec2994c1be4"
server_key_iv: "36b70f49d9c5ff65e9f11d123e5d7ce9"
server_key_version: "encryptionkey@0"
I20260927 16:32:27.446945 4507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:32:27.449469 4522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:27.450645 4507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:32:27.450779 4507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc"
format_stamp: "Formatted at 2026-09-27 16:32:27 on dist-test-slave-grzf"
server_key: "e2a2a0a4134ba36a7b799ec2994c1be4"
server_key_iv: "36b70f49d9c5ff65e9f11d123e5d7ce9"
server_key_version: "encryptionkey@0"
I20260927 16:32:27.450886 4507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:27.472921 4507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:27.475736 4507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:27.475942 4507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:27.482774 4507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46045
I20260927 16:32:27.482800 4574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46045 every 8 connection(s)
I20260927 16:32:27.483675 4507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:27.486207 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:27.486538 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4507
I20260927 16:32:27.486683 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:27.486949 15734 external_mini_cluster.cc:1428] Setting key c8888a8e396189405153b4e8b36631ce
I20260927 16:32:27.493099 4575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc: Bootstrap starting.
I20260927 16:32:27.495561 4575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc: Neither blocks nor log segments found. Creating new log.
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 16:32:27.496307 4575 log.cc:824] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:27.498586 4575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc: No bootstrap required, opened a new log
I20260927 16:32:27.501762 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:27.488351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:42891 (local address 127.0.0.1:46045)
0927 16:32:27.488763 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:27.488781 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:27.488823 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:32:27.489524 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:27.489546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:27.489975 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:27.490326 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:27.490986 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:27.492475 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:27.493348 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:27.494002 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:27.494165 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:27.496696 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:27.496709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:27.496721 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:27.496743 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:27.498657 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:27.499129 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:27.499135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:27.499141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:27.499272 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:27.499631 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:27.499635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:27.499636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:27.500101 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:27.500300 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:27.500543 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 16:32:27.500800 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
I20260927 16:32:27.502969 4575 raft_consensus.cc:348] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260927 16:32:27.503221 4575 raft_consensus.cc:374] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:27.503279 4575 raft_consensus.cc:729] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acabad1dfcf049a784bf8c5bbb0ff7fc, State: Initialized, Role: FOLLOWER
I20260927 16:32:27.503757 4575 consensus_queue.cc:260] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260927 16:32:27.503928 4575 raft_consensus.cc:388] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:27.503993 4575 raft_consensus.cc:482] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:27.504089 4575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:27.504752 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:27.505120 4575 raft_consensus.cc:504] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260927 16:32:27.505439 4575 leader_election.cc:302] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acabad1dfcf049a784bf8c5bbb0ff7fc; no voters:
I20260927 16:32:27.505811 4575 leader_election.cc:288] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:27.505904 4580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:27.506117 4580 raft_consensus.cc:686] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [term 1 LEADER]: Becoming Leader. State: Replica: acabad1dfcf049a784bf8c5bbb0ff7fc, State: Running, Role: LEADER
I20260927 16:32:27.506394 4580 consensus_queue.cc:237] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260927 16:32:27.507084 4575 sys_catalog.cc:564] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:27.507597 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260927 16:32:27.507761 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:27.508010 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader acabad1dfcf049a784bf8c5bbb0ff7fc. Latest consensus state: current_term: 1 leader_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acabad1dfcf049a784bf8c5bbb0ff7fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260927 16:32:27.508196 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:27.508512 4591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:27.511979 4591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:27.515810 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:27.506759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:39675 (local address 127.0.0.1:46045)
0927 16:32:27.506872 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:27.506875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:27.507194 (+ 319us) server_negotiation.cc:407] Connection header received
0927 16:32:27.507389 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:27.507395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:27.507441 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:27.507511 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:27.508413 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:27.509155 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:27.510277 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:27.510520 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:27.510636 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:27.512273 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:27.512276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:27.512281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:27.512305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:27.513802 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:27.514356 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:27.514360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:27.514361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:27.514454 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:27.514835 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:27.514838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:27.514840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:27.515029 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:27.515150 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:27.515587 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 16:32:27.515678 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:32:27.517953 4591 catalog_manager.cc:1347] Generated new cluster ID: d2b6129a38b94dd7a845faa79d5dfa52
I20260927 16:32:27.518015 4591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:27.555605 4591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:27.556587 4591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:27.560779 4591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acabad1dfcf049a784bf8c5bbb0ff7fc: Generated new TSK 0
I20260927 16:32:27.561439 4591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:27.572723 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4507
W20260927 16:32:27.577821 4492 connection.cc:535] client connection to 127.0.0.1:46045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46045): Connection reset by peer (error 104)
2026-09-27T16:32:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4606](info): set up 2 sockets
Sep 27 16:32:27 dist-test-slave-grzf krb5kdc[4606](info): commencing operation
krb5kdc: starting...
W20260927 16:32:28.603726 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:28 dist-test-slave-grzf krb5kdc[4606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:28.720359 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:28.824770 4619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:28.825065 4619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:28.825145 4619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:28.829077 4619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:28.829167 4619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:28.829206 4619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:28.829267 4619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:28.829305 4619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:28.834451 4619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:28.835489 4619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:28.836458 4619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:28.844631 4627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:28.844641 4624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:28.844650 4625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:28.844882 4619 server_base.cc:1034] running on GCE node
I20260927 16:32:28.845508 4619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:28.846570 4619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:28.847741 4619 hybrid_clock.cc:648] HybridClock initialized: now 1790526748847712 us; error 42 us; skew 500 ppm
Sep 27 16:32:28 dist-test-slave-grzf krb5kdc[4606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:28.850282 4619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:28.851130 4619 webserver.cc:460] Webserver started at http://127.0.0.1:40485/ using document root <none> and password file <none>
I20260927 16:32:28.851784 4619 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:28.851845 4619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:28.852001 4619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:28.853892 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "ec26998fc94d407b8bfac03b42d94554"
format_stamp: "Formatted at 2026-09-27 16:32:28 on dist-test-slave-grzf"
server_key: "57cdf3d0a9017da425c752dc06b5d3f3"
server_key_iv: "6fa0d5f0878b0511a04df6ceafae7faa"
server_key_version: "encryptionkey@0"
I20260927 16:32:28.854391 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "ec26998fc94d407b8bfac03b42d94554"
format_stamp: "Formatted at 2026-09-27 16:32:28 on dist-test-slave-grzf"
server_key: "57cdf3d0a9017da425c752dc06b5d3f3"
server_key_iv: "6fa0d5f0878b0511a04df6ceafae7faa"
server_key_version: "encryptionkey@0"
I20260927 16:32:28.857937 4619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:32:28.860132 4634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:28.861280 4619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:28.861409 4619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "ec26998fc94d407b8bfac03b42d94554"
format_stamp: "Formatted at 2026-09-27 16:32:28 on dist-test-slave-grzf"
server_key: "57cdf3d0a9017da425c752dc06b5d3f3"
server_key_iv: "6fa0d5f0878b0511a04df6ceafae7faa"
server_key_version: "encryptionkey@0"
I20260927 16:32:28.861505 4619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:28.875840 4619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:28.878983 4619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:28.879185 4619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:28.885818 4619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39945
I20260927 16:32:28.885877 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39945 every 8 connection(s)
I20260927 16:32:28.886754 4619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:28.889392 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:28.894641 4687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554: Bootstrap starting.
I20260927 16:32:28.895604 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4619
I20260927 16:32:28.895750 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:28.895998 15734 external_mini_cluster.cc:1428] Setting key 7de7d9fa832b578e0fed78f62c9ff9d9
I20260927 16:32:28.897290 4687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:28.897981 4687 log.cc:824] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:28.899927 4687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554: No bootstrap required, opened a new log
I20260927 16:32:28.904160 4687 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260927 16:32:28.904368 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:28.897060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:34377 (local address 127.0.0.1:39945)
0927 16:32:28.897490 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:28.897504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:28.897537 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:28.898183 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:28.898201 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:28.898515 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:28.898784 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:28.899084 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:28.900134 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:28.901314 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:28.901759 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:28.901875 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:28.902726 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:28.902731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:28.902742 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:28.903103 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:28.903151 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:28.903172 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:28.903277 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:28.903500 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 16:32:28.903677 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":130,"threads_started":1}
I20260927 16:32:28.904385 4687 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:28.904537 4687 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec26998fc94d407b8bfac03b42d94554, State: Initialized, Role: FOLLOWER
I20260927 16:32:28.905012 4687 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260927 16:32:28.905182 4687 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:28.905242 4687 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:28.905331 4687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:28.906194 4687 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260927 16:32:28.906529 4687 leader_election.cc:302] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec26998fc94d407b8bfac03b42d94554; no voters:
I20260927 16:32:28.906888 4687 leader_election.cc:288] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:28.906936 4692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:28.907114 4692 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [term 1 LEADER]: Becoming Leader. State: Replica: ec26998fc94d407b8bfac03b42d94554, State: Running, Role: LEADER
I20260927 16:32:28.907269 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:28.907464 4692 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } }
I20260927 16:32:28.908063 4687 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:28.908923 4693 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec26998fc94d407b8bfac03b42d94554. Latest consensus state: current_term: 1 leader_uuid: "ec26998fc94d407b8bfac03b42d94554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260927 16:32:28.909052 4693 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:28.909412 4705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:28.909381 4694 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec26998fc94d407b8bfac03b42d94554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec26998fc94d407b8bfac03b42d94554" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39945 } } }
I20260927 16:32:28.909464 4694 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:28.912494 4705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:28.915894 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:28.910370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:60687 (local address 127.0.0.1:39945)
0927 16:32:28.910488 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:28.910492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:28.911147 (+ 655us) server_negotiation.cc:407] Connection header received
0927 16:32:28.911505 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:28.911520 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:28.911572 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:28.911665 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:28.911955 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:28.912657 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:28.913951 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:28.914295 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:28.914402 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:28.915417 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:28.915421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:28.915424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:28.915436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:28.915464 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:28.915481 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:28.915592 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:28.915734 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 16:32:28.915766 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:28.918172 4705 catalog_manager.cc:1347] Generated new cluster ID: 453c82f405524edab3e040d55294c386
I20260927 16:32:28.918229 4705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:28.928782 4705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:28.929991 4705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:28.942293 4705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec26998fc94d407b8bfac03b42d94554: Generated new TSK 0
I20260927 16:32:28.943120 4705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:28.951953 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4619
W20260927 16:32:28.957278 4604 connection.cc:535] client connection to 127.0.0.1:39945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39945): Connection reset by peer (error 104)
2026-09-27T16:32:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:29 dist-test-slave-grzf krb5kdc[4718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:29 dist-test-slave-grzf krb5kdc[4718](info): set up 2 sockets
Sep 27 16:32:29 dist-test-slave-grzf krb5kdc[4718](info): commencing operation
krb5kdc: starting...
W20260927 16:32:29.983564 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:30 dist-test-slave-grzf krb5kdc[4718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 16:32:30.101666 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:30.210247 4731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:30.210515 4731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:30.210588 4731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:30.214679 4731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:30.214762 4731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:30.214802 4731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:30.214838 4731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:30.214869 4731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:30.220017 4731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:30.221010 4731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:30.221971 4731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:30.229998 4737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:30.229993 4739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:30.230082 4731 server_base.cc:1034] running on GCE node
W20260927 16:32:30.229993 4736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:30.230830 4731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:30.231945 4731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:30.233134 4731 hybrid_clock.cc:648] HybridClock initialized: now 1790526750233122 us; error 24 us; skew 500 ppm
Sep 27 16:32:30 dist-test-slave-grzf krb5kdc[4718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:30.241797 4731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 16:32:30.242726 4731 webserver.cc:460] Webserver started at http://127.0.0.1:38759/ using document root <none> and password file <none>
I20260927 16:32:30.243283 4731 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:30.243374 4731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:30.243616 4731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:30.245597 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "95856c1a922f4dee8d8a71c3e7b59098"
format_stamp: "Formatted at 2026-09-27 16:32:30 on dist-test-slave-grzf"
server_key: "6be81ae61bb8f41f37557a142c7cdbaf"
server_key_iv: "e903a56c34403eed5dabb4687bf02171"
server_key_version: "encryptionkey@0"
I20260927 16:32:30.246097 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "95856c1a922f4dee8d8a71c3e7b59098"
format_stamp: "Formatted at 2026-09-27 16:32:30 on dist-test-slave-grzf"
server_key: "6be81ae61bb8f41f37557a142c7cdbaf"
server_key_iv: "e903a56c34403eed5dabb4687bf02171"
server_key_version: "encryptionkey@0"
I20260927 16:32:30.249661 4731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 16:32:30.251896 4746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:30.253330 4731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:30.253455 4731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "95856c1a922f4dee8d8a71c3e7b59098"
format_stamp: "Formatted at 2026-09-27 16:32:30 on dist-test-slave-grzf"
server_key: "6be81ae61bb8f41f37557a142c7cdbaf"
server_key_iv: "e903a56c34403eed5dabb4687bf02171"
server_key_version: "encryptionkey@0"
I20260927 16:32:30.253571 4731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:30.257881 4731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:30.260708 4731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:30.260844 4731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:30.267431 4731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45035
I20260927 16:32:30.267454 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45035 every 8 connection(s)
I20260927 16:32:30.268283 4731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:30.270803 4799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:30.276052 4799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098: Bootstrap starting.
I20260927 16:32:30.277206 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4731
I20260927 16:32:30.277343 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:30.277633 15734 external_mini_cluster.cc:1428] Setting key 41c230cc3192de351d7f503e0656f185
I20260927 16:32:30.278631 4799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:30.279536 4799 log.cc:824] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:30.281595 4799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098: No bootstrap required, opened a new log
I20260927 16:32:30.282795 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:30.278643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:59299 (local address 127.0.0.1:45035)
0927 16:32:30.279120 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:30.279136 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:30.279176 (+ 40us) server_negotiation.cc:407] Connection header received
0927 16:32:30.280009 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:30.280027 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:30.280341 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:30.280681 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:30.281331 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:30.281337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:30.281350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:30.281361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:30.281418 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:30.281444 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:30.281550 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:30.281649 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 16:32:30.281948 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260927 16:32:30.285410 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:30.285796 4799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260927 16:32:30.286012 4799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:30.286088 4799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95856c1a922f4dee8d8a71c3e7b59098, State: Initialized, Role: FOLLOWER
I20260927 16:32:30.286518 4799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260927 16:32:30.286676 4799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:30.286748 4799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:30.286847 4799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:30.287894 4799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260927 16:32:30.288270 4799 leader_election.cc:302] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95856c1a922f4dee8d8a71c3e7b59098; no voters:
I20260927 16:32:30.288615 4799 leader_election.cc:288] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:30.288712 4809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:30.288911 4809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [term 1 LEADER]: Becoming Leader. State: Replica: 95856c1a922f4dee8d8a71c3e7b59098, State: Running, Role: LEADER
I20260927 16:32:30.289202 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:30.287477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:60561 (local address 127.0.0.1:45035)
0927 16:32:30.287601 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:30.287605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:30.287767 (+ 162us) server_negotiation.cc:407] Connection header received
0927 16:32:30.287896 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:30.287898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:30.287924 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:30.287986 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:30.288761 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:30.288764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:30.288766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:30.288779 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:30.288808 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:30.288825 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:30.288885 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:30.289052 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 16:32:30.289100 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 16:32:30.289232 4809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } }
I20260927 16:32:30.289877 4799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:30.290910 4810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95856c1a922f4dee8d8a71c3e7b59098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } } }
I20260927 16:32:30.291045 4810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:30.291376 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95856c1a922f4dee8d8a71c3e7b59098. Latest consensus state: current_term: 1 leader_uuid: "95856c1a922f4dee8d8a71c3e7b59098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95856c1a922f4dee8d8a71c3e7b59098" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45035 } } }
I20260927 16:32:30.291469 4818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:30.291531 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:30.294590 4818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:30.300205 4818 catalog_manager.cc:1347] Generated new cluster ID: a3e8a1a553a0488ba0e1fb1b65401088
I20260927 16:32:30.300275 4818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:30.307034 4818 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:30.307329 4818 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:30.319067 4818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95856c1a922f4dee8d8a71c3e7b59098: Generated new TSK 0
I20260927 16:32:30.319805 4818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:30.327764 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4731
2026-09-27T16:32:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:30 dist-test-slave-grzf krb5kdc[4830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:30 dist-test-slave-grzf krb5kdc[4830](info): set up 2 sockets
Sep 27 16:32:30 dist-test-slave-grzf krb5kdc[4830](info): commencing operation
krb5kdc: starting...
W20260927 16:32:31.355727 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:31.469678 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44561
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:31.574301 4843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:31.574615 4843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:31.574703 4843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:31.578778 4843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:31.578855 4843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:31.578892 4843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:31.578948 4843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:31.578986 4843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:31.584025 4843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:44561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:31.585001 4843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:31.585987 4843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:31.594125 4851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:31.594134 4848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:31.594291 4849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:31.594347 4843 server_base.cc:1034] running on GCE node
I20260927 16:32:31.594898 4843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:31.595944 4843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:31.597112 4843 hybrid_clock.cc:648] HybridClock initialized: now 1790526751597086 us; error 38 us; skew 500 ppm
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526751, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:31.599659 4843 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:31.600451 4843 webserver.cc:460] Webserver started at http://127.15.93.190:40501/ using document root <none> and password file <none>
I20260927 16:32:31.600976 4843 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:31.601085 4843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:31.601297 4843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:31.603202 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "6ab7f32d22484016bb53035366a534e7"
format_stamp: "Formatted at 2026-09-27 16:32:31 on dist-test-slave-grzf"
server_key: "b79eda77fe33b61d23af3b3c1cd63a89"
server_key_iv: "2518c79db46c29a772b47cb7d1273075"
server_key_version: "encryptionkey@0"
I20260927 16:32:31.603793 4843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "6ab7f32d22484016bb53035366a534e7"
format_stamp: "Formatted at 2026-09-27 16:32:31 on dist-test-slave-grzf"
server_key: "b79eda77fe33b61d23af3b3c1cd63a89"
server_key_iv: "2518c79db46c29a772b47cb7d1273075"
server_key_version: "encryptionkey@0"
I20260927 16:32:31.607208 4843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 16:32:31.609452 4858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:31.610598 4843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:31.610729 4843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "6ab7f32d22484016bb53035366a534e7"
format_stamp: "Formatted at 2026-09-27 16:32:31 on dist-test-slave-grzf"
server_key: "b79eda77fe33b61d23af3b3c1cd63a89"
server_key_iv: "2518c79db46c29a772b47cb7d1273075"
server_key_version: "encryptionkey@0"
I20260927 16:32:31.610839 4843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:31.626914 4843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:31.629952 4843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:31.630172 4843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:31.637241 4843 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:44561
I20260927 16:32:31.637250 4910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:44561 every 8 connection(s)
I20260927 16:32:31.638168 4843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:31.640880 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:31.644961 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4843
I20260927 16:32:31.645114 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:31.645381 15734 external_mini_cluster.cc:1428] Setting key 9db4f05dd4199c370985111636fc10a3
I20260927 16:32:31.646473 4911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7: Bootstrap starting.
I20260927 16:32:31.648730 4911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:31.649581 4911 log.cc:824] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:31.651630 4911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7: No bootstrap required, opened a new log
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:31.655754 4911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } }
I20260927 16:32:31.655980 4911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:31.656046 4911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab7f32d22484016bb53035366a534e7, State: Initialized, Role: FOLLOWER
I20260927 16:32:31.656526 4911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } }
I20260927 16:32:31.656715 4911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:31.656782 4911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:31.656883 4911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:31.657889 4911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } }
I20260927 16:32:31.658262 4911 leader_election.cc:302] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ab7f32d22484016bb53035366a534e7; no voters:
I20260927 16:32:31.658593 4911 leader_election.cc:288] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:31.658682 4916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:31.658880 4916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab7f32d22484016bb53035366a534e7, State: Running, Role: LEADER
I20260927 16:32:31.659260 4916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } }
I20260927 16:32:31.659890 4911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:31.660341 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab7f32d22484016bb53035366a534e7. Latest consensus state: current_term: 1 leader_uuid: "6ab7f32d22484016bb53035366a534e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } } }
I20260927 16:32:31.660444 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:31.646776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40115 (local address 127.15.93.190:44561)
0927 16:32:31.647151 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:31.647175 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:31.647208 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:31.647867 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:31.647884 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:31.648206 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:31.648468 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:31.649378 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:31.650399 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:31.651551 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:31.652067 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:31.652194 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:31.654927 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:31.654932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:31.654943 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:31.654967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:31.657042 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:31.657524 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:31.657530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:31.657537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:31.657677 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:31.657978 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:31.657981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:31.657982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:31.658499 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:31.658712 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:31.659289 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 16:32:31.659527 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260927 16:32:31.660449 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:31.661139 4925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:31.662804 4917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ab7f32d22484016bb53035366a534e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ab7f32d22484016bb53035366a534e7" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 44561 } } }
I20260927 16:32:31.662899 4917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:31.664099 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:31.664199 4925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:31.670212 4925 catalog_manager.cc:1347] Generated new cluster ID: 025dda8c37b14776b69a54074969ed9c
I20260927 16:32:31.670271 4925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:31.676015 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:31.666428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55025 (local address 127.15.93.190:44561)
0927 16:32:31.666555 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:31.666559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:31.666934 (+ 375us) server_negotiation.cc:407] Connection header received
0927 16:32:31.667024 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:31.667027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:31.667079 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:31.667165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:31.667945 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:31.668752 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:31.669888 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:31.670304 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:31.670411 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:31.672143 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:31.672147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:31.672150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:31.672174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:31.673895 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:31.674510 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:31.674514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:31.674516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:31.674600 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:31.675005 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:31.675008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:31.675010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:31.675216 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:31.675328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:31.675776 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 16:32:31.675869 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:32:31.689186 4925 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:31.690244 4925 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:31.696972 4925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ab7f32d22484016bb53035366a534e7: Generated new TSK 0
I20260927 16:32:31.697636 4925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:31.702606 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4843
W20260927 16:32:31.708194 4828 connection.cc:535] client connection to 127.15.93.190:44561 recv error: Network error: failed to read from TLS socket (remote: 127.15.93.190:44561): Connection reset by peer (error 104)
2026-09-27T16:32:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4942](info): set up 2 sockets
Sep 27 16:32:31 dist-test-slave-grzf krb5kdc[4942](info): commencing operation
krb5kdc: starting...
W20260927 16:32:32.743647 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:32 dist-test-slave-grzf krb5kdc[4942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:32.863324 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43135
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:32.971949 4955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:32.972219 4955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:32.972294 4955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:32.976254 4955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:32.976342 4955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:32.976382 4955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:32.976418 4955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:32.976465 4955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:32.981593 4955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:43135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:32.982561 4955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:32.983541 4955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:32.991632 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:32.991639 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:32.991873 4955 server_base.cc:1034] running on GCE node
W20260927 16:32:32.991668 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:32.992502 4955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:32.993520 4955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:32.994688 4955 hybrid_clock.cc:648] HybridClock initialized: now 1790526752994669 us; error 30 us; skew 500 ppm
Sep 27 16:32:32 dist-test-slave-grzf krb5kdc[4942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526752, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:32.997236 4955 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:32.998087 4955 webserver.cc:460] Webserver started at http://127.15.93.190:45903/ using document root <none> and password file <none>
I20260927 16:32:32.998668 4955 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:32.998728 4955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:32.998894 4955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:33.000869 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "84522267c18d4cdfbcb669e84927747a"
format_stamp: "Formatted at 2026-09-27 16:32:32 on dist-test-slave-grzf"
server_key: "378db9d24214b9474402e1b05649687c"
server_key_iv: "46c18347d5ab9400ada401151f4a5fe7"
server_key_version: "encryptionkey@0"
I20260927 16:32:33.001362 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "84522267c18d4cdfbcb669e84927747a"
format_stamp: "Formatted at 2026-09-27 16:32:32 on dist-test-slave-grzf"
server_key: "378db9d24214b9474402e1b05649687c"
server_key_iv: "46c18347d5ab9400ada401151f4a5fe7"
server_key_version: "encryptionkey@0"
I20260927 16:32:33.004863 4955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:33.007066 4970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:33.008265 4955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:32:33.008365 4955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "84522267c18d4cdfbcb669e84927747a"
format_stamp: "Formatted at 2026-09-27 16:32:32 on dist-test-slave-grzf"
server_key: "378db9d24214b9474402e1b05649687c"
server_key_iv: "46c18347d5ab9400ada401151f4a5fe7"
server_key_version: "encryptionkey@0"
I20260927 16:32:33.008445 4955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:33.046474 4955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:33.049386 4955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:33.049546 4955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:33.056509 4955 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:43135
I20260927 16:32:33.056526 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:43135 every 8 connection(s)
I20260927 16:32:33.057425 4955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:33.059554 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 4955
I20260927 16:32:33.059695 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:33.059988 15734 external_mini_cluster.cc:1428] Setting key 1da793f8683e936d6e28cb9a7c634256
I20260927 16:32:33.060223 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:33.066224 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a: Bootstrap starting.
I20260927 16:32:33.068588 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:33.061301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43407 (local address 127.15.93.190:43135)
0927 16:32:33.061722 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:33.061737 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:33.061770 (+ 33us) server_negotiation.cc:407] Connection header received
0927 16:32:33.062397 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:33.062414 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:33.062742 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:33.063067 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:33.063512 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:33.064465 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:33.065350 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:33.065828 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:33.065999 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:33.066925 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:33.066930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:33.066942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:33.067320 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:33.067418 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:33.067449 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:33.067597 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:33.067758 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 16:32:33.067897 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":146,"threads_started":1}
I20260927 16:32:33.068614 5023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:33.069362 5023 log.cc:824] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:33.071306 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:33.071334 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a: No bootstrap required, opened a new log
I20260927 16:32:33.075524 5023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } }
I20260927 16:32:33.075830 5023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:33.075929 5023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84522267c18d4cdfbcb669e84927747a, State: Initialized, Role: FOLLOWER
I20260927 16:32:33.076416 5023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } }
I20260927 16:32:33.076608 5023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:33.076678 5023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:33.076776 5023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:33.077675 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:33.073253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43569 (local address 127.15.93.190:43135)
0927 16:32:33.073361 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:33.073364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:33.073634 (+ 270us) server_negotiation.cc:407] Connection header received
0927 16:32:33.073771 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:33.073773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:33.073808 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:33.073864 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:33.074166 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:33.074665 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:33.075776 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:33.076124 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:33.076237 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:33.077201 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:33.077204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:33.077206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:33.077216 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:33.077239 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:33.077249 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:33.077315 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:33.077483 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 16:32:33.077540 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:32:33.077965 5023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } }
I20260927 16:32:33.078346 5023 leader_election.cc:302] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84522267c18d4cdfbcb669e84927747a; no voters:
I20260927 16:32:33.078691 5023 leader_election.cc:288] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:33.078789 5033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:33.078999 5033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [term 1 LEADER]: Becoming Leader. State: Replica: 84522267c18d4cdfbcb669e84927747a, State: Running, Role: LEADER
I20260927 16:32:33.079344 5033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } }
I20260927 16:32:33.079923 5023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:33.081017 5035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84522267c18d4cdfbcb669e84927747a. Latest consensus state: current_term: 1 leader_uuid: "84522267c18d4cdfbcb669e84927747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } } }
I20260927 16:32:33.081019 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84522267c18d4cdfbcb669e84927747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84522267c18d4cdfbcb669e84927747a" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 43135 } } }
I20260927 16:32:33.081199 5035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:33.081226 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:33.081527 5042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:33.085237 5042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:33.090304 5042 catalog_manager.cc:1347] Generated new cluster ID: 1c85e4cb15f9474b8b094ff0ff6e920c
I20260927 16:32:33.090365 5042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:33.105237 5042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:33.106186 5042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:33.115239 5042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84522267c18d4cdfbcb669e84927747a: Generated new TSK 0
I20260927 16:32:33.115912 5042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:33.122620 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 4955
W20260927 16:32:33.127942 4940 connection.cc:535] client connection to 127.15.93.190:43135 recv error: Network error: failed to read from TLS socket (remote: 127.15.93.190:43135): Connection reset by peer (error 104)
2026-09-27T16:32:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:33 dist-test-slave-grzf krb5kdc[5054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:33 dist-test-slave-grzf krb5kdc[5054](info): set up 2 sockets
Sep 27 16:32:33 dist-test-slave-grzf krb5kdc[5054](info): commencing operation
krb5kdc: starting...
W20260927 16:32:34.163684 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:34 dist-test-slave-grzf krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:34.282224 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40537
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:34.389135 5067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:34.389375 5067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:34.389416 5067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:34.393211 5067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:34.393257 5067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:34.393276 5067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:34.393292 5067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:34.393307 5067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:34.398257 5067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:40537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:34.399163 5067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:34.400094 5067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:34.408126 5072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:34.408221 5067 server_base.cc:1034] running on GCE node
W20260927 16:32:34.408097 5073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:34.408092 5075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:34.408910 5067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:34.409929 5067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:34.411111 5067 hybrid_clock.cc:648] HybridClock initialized: now 1790526754411093 us; error 30 us; skew 500 ppm
Sep 27 16:32:34 dist-test-slave-grzf krb5kdc[5054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526754, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:34.413729 5067 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:34.414575 5067 webserver.cc:460] Webserver started at http://127.15.93.190:42635/ using document root <none> and password file <none>
I20260927 16:32:34.415105 5067 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:34.415192 5067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:34.415470 5067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:34.417497 5067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "ff59a1464787495e84e30198f015d64e"
format_stamp: "Formatted at 2026-09-27 16:32:34 on dist-test-slave-grzf"
server_key: "e96bf2fa5c2e07bccb241f4a6bb79d31"
server_key_iv: "4a85c0556dad71e9cf95d93fb06663e9"
server_key_version: "encryptionkey@0"
I20260927 16:32:34.418061 5067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "ff59a1464787495e84e30198f015d64e"
format_stamp: "Formatted at 2026-09-27 16:32:34 on dist-test-slave-grzf"
server_key: "e96bf2fa5c2e07bccb241f4a6bb79d31"
server_key_iv: "4a85c0556dad71e9cf95d93fb06663e9"
server_key_version: "encryptionkey@0"
I20260927 16:32:34.421700 5067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:34.424295 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:34.425588 5067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:34.425735 5067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "ff59a1464787495e84e30198f015d64e"
format_stamp: "Formatted at 2026-09-27 16:32:34 on dist-test-slave-grzf"
server_key: "e96bf2fa5c2e07bccb241f4a6bb79d31"
server_key_iv: "4a85c0556dad71e9cf95d93fb06663e9"
server_key_version: "encryptionkey@0"
I20260927 16:32:34.425848 5067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:34.430316 5067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:34.433178 5067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:34.433357 5067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:34.439874 5067 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:40537
I20260927 16:32:34.439929 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:40537 every 8 connection(s)
I20260927 16:32:34.440773 5067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:34.443347 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:34.447674 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5067
I20260927 16:32:34.447813 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:34.448076 15734 external_mini_cluster.cc:1428] Setting key c341d8d076042d96e10e3560419db71b
I20260927 16:32:34.448822 5135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e: Bootstrap starting.
I20260927 16:32:34.451397 5135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:34.452289 5135 log.cc:824] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:34.453476 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:34.449386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38209 (local address 127.15.93.190:40537)
0927 16:32:34.449776 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:34.449795 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:34.449842 (+ 47us) server_negotiation.cc:407] Connection header received
0927 16:32:34.450668 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:34.450702 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:34.451037 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:34.451428 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:34.452180 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:34.452186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:34.452200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:34.452212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:34.452272 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:34.452301 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:34.452419 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:34.452568 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 16:32:34.452723 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260927 16:32:34.454201 5135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e: No bootstrap required, opened a new log
I20260927 16:32:34.456132 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:34.457943 5135 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } }
I20260927 16:32:34.458251 5135 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:34.458307 5135 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff59a1464787495e84e30198f015d64e, State: Initialized, Role: FOLLOWER
I20260927 16:32:34.458765 5135 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } }
I20260927 16:32:34.458940 5135 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:34.459013 5135 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:34.459123 5135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:34.459717 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:34.458062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44927 (local address 127.15.93.190:40537)
0927 16:32:34.458188 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:34.458192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:34.458408 (+ 216us) server_negotiation.cc:407] Connection header received
0927 16:32:34.458474 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:34.458476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:34.458511 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:34.458602 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:34.459339 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:34.459346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:34.459352 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:34.459388 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:34.459414 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:34.459428 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:34.459517 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:34.459575 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 16:32:34.459610 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 16:32:34.460203 5135 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } }
I20260927 16:32:34.460544 5135 leader_election.cc:302] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff59a1464787495e84e30198f015d64e; no voters:
I20260927 16:32:34.460848 5135 leader_election.cc:288] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:34.460965 5145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:34.461200 5145 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [term 1 LEADER]: Becoming Leader. State: Replica: ff59a1464787495e84e30198f015d64e, State: Running, Role: LEADER
I20260927 16:32:34.461516 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } }
I20260927 16:32:34.462026 5135 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:34.463366 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff59a1464787495e84e30198f015d64e. Latest consensus state: current_term: 1 leader_uuid: "ff59a1464787495e84e30198f015d64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } } }
I20260927 16:32:34.463511 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:34.463804 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff59a1464787495e84e30198f015d64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff59a1464787495e84e30198f015d64e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 40537 } } }
I20260927 16:32:34.463955 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:34.464251 5155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:34.466696 5155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:34.471752 5155 catalog_manager.cc:1347] Generated new cluster ID: 29ee5e77ccf34a7ba26dac471cd1d0a5
I20260927 16:32:34.471812 5155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:34.481833 5155 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:34.482142 5155 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:34.487568 5155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff59a1464787495e84e30198f015d64e: Generated new TSK 0
I20260927 16:32:34.488263 5155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:34.491677 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5067
2026-09-27T16:32:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:34 dist-test-slave-grzf krb5kdc[5166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:34 dist-test-slave-grzf krb5kdc[5166](info): set up 2 sockets
Sep 27 16:32:34 dist-test-slave-grzf krb5kdc[5166](info): commencing operation
krb5kdc: starting...
W20260927 16:32:35.515686 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:35.630141 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:45165
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:35.736840 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:35.737082 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:35.737120 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:35.740971 5179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:35.741024 5179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:35.741043 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:35.741060 5179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:35.741074 5179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:35.746034 5179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:35.746949 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:35.747877 5179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:32:35.755796 5179 server_base.cc:1034] running on GCE node
W20260927 16:32:35.755769 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:35.755762 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:35.755762 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:35.756565 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:35.757624 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:35.758796 5179 hybrid_clock.cc:648] HybridClock initialized: now 1790526755758780 us; error 34 us; skew 500 ppm
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526755, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:35.766865 5179 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:35.767764 5179 webserver.cc:460] Webserver started at http://127.15.93.190:46519/ using document root <none> and password file <none>
I20260927 16:32:35.768338 5179 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:35.768420 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:35.768676 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:35.770704 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "2ce27cf10b5d4547bf3418e86b9214d8"
format_stamp: "Formatted at 2026-09-27 16:32:35 on dist-test-slave-grzf"
server_key: "29dda451305c019a223b570f025ba5c1"
server_key_iv: "f531f56f5d840c935abbe143f288c22a"
server_key_version: "encryptionkey@0"
I20260927 16:32:35.771260 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "2ce27cf10b5d4547bf3418e86b9214d8"
format_stamp: "Formatted at 2026-09-27 16:32:35 on dist-test-slave-grzf"
server_key: "29dda451305c019a223b570f025ba5c1"
server_key_iv: "f531f56f5d840c935abbe143f288c22a"
server_key_version: "encryptionkey@0"
I20260927 16:32:35.774857 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 16:32:35.777184 5194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:35.778322 5179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:35.778470 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "2ce27cf10b5d4547bf3418e86b9214d8"
format_stamp: "Formatted at 2026-09-27 16:32:35 on dist-test-slave-grzf"
server_key: "29dda451305c019a223b570f025ba5c1"
server_key_iv: "f531f56f5d840c935abbe143f288c22a"
server_key_version: "encryptionkey@0"
I20260927 16:32:35.778589 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:35.787308 5179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:35.789945 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:35.790125 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:35.796739 5179 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:45165
I20260927 16:32:35.796784 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:45165 every 8 connection(s)
I20260927 16:32:35.797856 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:35.800508 5247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:35.805557 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5179
I20260927 16:32:35.805672 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8: Bootstrap starting.
I20260927 16:32:35.805714 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:35.805969 15734 external_mini_cluster.cc:1428] Setting key 03f78e7b1a762bb008117d2528718feb
I20260927 16:32:35.808307 5247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:35.809156 5247 log.cc:824] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:35.811246 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8: No bootstrap required, opened a new log
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:35.815178 5247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } }
I20260927 16:32:35.815483 5247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:35.815557 5247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce27cf10b5d4547bf3418e86b9214d8, State: Initialized, Role: FOLLOWER
I20260927 16:32:35.816012 5247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } }
I20260927 16:32:35.816193 5247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:35.816274 5247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:35.816377 5247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:35.817538 5247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } }
I20260927 16:32:35.817889 5247 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce27cf10b5d4547bf3418e86b9214d8; no voters:
I20260927 16:32:35.818207 5247 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:35.818296 5252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:35.818506 5252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce27cf10b5d4547bf3418e86b9214d8, State: Running, Role: LEADER
I20260927 16:32:35.818886 5252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } }
I20260927 16:32:35.819404 5247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:35.820120 5254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce27cf10b5d4547bf3418e86b9214d8. Latest consensus state: current_term: 1 leader_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } } }
I20260927 16:32:35.820240 5254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:35.820930 5253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce27cf10b5d4547bf3418e86b9214d8" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 45165 } } }
I20260927 16:32:35.821074 5253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:35.821375 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:35.807229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59995 (local address 127.15.93.190:45165)
0927 16:32:35.807687 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:35.807705 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:35.807744 (+ 39us) server_negotiation.cc:407] Connection header received
0927 16:32:35.808600 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:35.808624 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:35.809032 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:35.809373 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:35.810222 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:35.811693 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:35.812537 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:35.813230 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:35.813384 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:35.815889 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:35.815894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:35.815909 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:35.815938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:35.817998 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:35.818555 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:35.818561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:35.818567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:35.818706 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:35.819019 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:35.819022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:35.819024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:35.819536 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:35.819727 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:35.820057 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 16:32:35.820385 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":159,"threads_started":1}
I20260927 16:32:35.821360 5260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:35.824733 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:35.825101 5260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:35.830646 5260 catalog_manager.cc:1347] Generated new cluster ID: 435bf0f264d741a499c074eb9a900981
I20260927 16:32:35.830714 5260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:35.839155 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:35.826730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45275 (local address 127.15.93.190:45165)
0927 16:32:35.826850 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:35.826855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:35.831070 (+ 4215us) server_negotiation.cc:407] Connection header received
0927 16:32:35.831192 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:35.831195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:35.831244 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:35.831330 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:35.832117 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:35.832896 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:35.833940 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:35.834328 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:35.834451 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:35.835915 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:35.835919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:35.835922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:35.835946 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:35.837569 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:35.838000 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:35.838003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:35.838005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:35.838095 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:35.838360 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:35.838363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:35.838365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:35.838559 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:35.838693 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:35.838892 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 16:32:35.838997 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:32:35.843464 5260 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:35.844349 5260 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:35.849325 5260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ce27cf10b5d4547bf3418e86b9214d8: Generated new TSK 0
I20260927 16:32:35.850121 5260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:35.853329 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5179
2026-09-27T16:32:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5278](info): set up 2 sockets
Sep 27 16:32:35 dist-test-slave-grzf krb5kdc[5278](info): commencing operation
krb5kdc: starting...
W20260927 16:32:36.883591 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:36 dist-test-slave-grzf krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:37.011929 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36993
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:40251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:37.120203 5291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:37.120452 5291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:37.120530 5291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:37.124524 5291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:37.124583 5291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:37.124601 5291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:37.124619 5291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:37.124634 5291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:37.129902 5291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:40251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:37.130955 5291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:37.131979 5291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:37.140187 5299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:37.140350 5291 server_base.cc:1034] running on GCE node
W20260927 16:32:37.140226 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:37.140475 5296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:37.141070 5291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:37.142110 5291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:37.143296 5291 hybrid_clock.cc:648] HybridClock initialized: now 1790526757143279 us; error 28 us; skew 500 ppm
Sep 27 16:32:37 dist-test-slave-grzf krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526757, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:37.145867 5291 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:37.146701 5291 webserver.cc:460] Webserver started at http://127.15.93.190:39817/ using document root <none> and password file <none>
I20260927 16:32:37.147236 5291 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:37.147318 5291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:37.147558 5291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:37.149541 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "2fb9859656fc46a8b511c65ceef2dbd3"
format_stamp: "Formatted at 2026-09-27 16:32:37 on dist-test-slave-grzf"
server_key: "c80af0f2ec9c368ba191e07a70b8b5bb"
server_key_iv: "8fbd929102de1160e2549307a38ae8c0"
server_key_version: "encryptionkey@0"
I20260927 16:32:37.150110 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "2fb9859656fc46a8b511c65ceef2dbd3"
format_stamp: "Formatted at 2026-09-27 16:32:37 on dist-test-slave-grzf"
server_key: "c80af0f2ec9c368ba191e07a70b8b5bb"
server_key_iv: "8fbd929102de1160e2549307a38ae8c0"
server_key_version: "encryptionkey@0"
I20260927 16:32:37.153757 5291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 16:32:37.156203 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:37.157372 5291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 16:32:37.157510 5291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "2fb9859656fc46a8b511c65ceef2dbd3"
format_stamp: "Formatted at 2026-09-27 16:32:37 on dist-test-slave-grzf"
server_key: "c80af0f2ec9c368ba191e07a70b8b5bb"
server_key_iv: "8fbd929102de1160e2549307a38ae8c0"
server_key_version: "encryptionkey@0"
I20260927 16:32:37.157634 5291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:37.176225 5291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:37.179119 5291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:37.179312 5291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:37.186105 5291 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36993
I20260927 16:32:37.186147 5358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36993 every 8 connection(s)
I20260927 16:32:37.187055 5291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:37.187652 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5291
I20260927 16:32:37.187796 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:37.188076 15734 external_mini_cluster.cc:1428] Setting key e220dad8c6b61ca18bbbca505a929f91
I20260927 16:32:37.190063 5359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:37.196398 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3: Bootstrap starting.
I20260927 16:32:37.196971 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:37.189409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47603 (local address 127.15.93.190:36993)
0927 16:32:37.189817 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:37.189835 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:37.189877 (+ 42us) server_negotiation.cc:407] Connection header received
0927 16:32:37.190564 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:37.190583 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:37.190962 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:37.191273 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:37.191633 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:37.192590 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:37.193772 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:37.194226 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:37.194340 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:37.195195 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:37.195200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:37.195211 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:37.195641 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:37.195692 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:37.195713 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:37.195787 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:37.195941 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 16:32:37.196162 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":143,"threads_started":1}
I20260927 16:32:37.198513 5359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:37.199291 5359 log.cc:824] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:37.199574 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:37.201192 5359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3: No bootstrap required, opened a new log
I20260927 16:32:37.205413 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:37.201556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40299 (local address 127.15.93.190:36993)
0927 16:32:37.201672 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:37.201676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:37.201808 (+ 132us) server_negotiation.cc:407] Connection header received
0927 16:32:37.201926 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:37.201928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:37.201961 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:37.202022 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:37.202281 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:37.202786 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:37.203987 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:37.204199 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:37.204275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:37.205005 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:37.205008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:37.205010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:37.205020 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:37.205040 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:37.205050 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:37.205105 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:37.205221 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 16:32:37.205261 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 16:32:37.205605 5359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } }
I20260927 16:32:37.205873 5359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:37.205937 5359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb9859656fc46a8b511c65ceef2dbd3, State: Initialized, Role: FOLLOWER
I20260927 16:32:37.206373 5359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } }
I20260927 16:32:37.206519 5359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:37.206583 5359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:37.206672 5359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:37.207552 5359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } }
I20260927 16:32:37.207866 5359 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb9859656fc46a8b511c65ceef2dbd3; no voters:
I20260927 16:32:37.208149 5359 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:37.208252 5369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:37.208463 5369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb9859656fc46a8b511c65ceef2dbd3, State: Running, Role: LEADER
I20260927 16:32:37.208758 5369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } }
I20260927 16:32:37.209297 5359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:37.210285 5370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } } }
I20260927 16:32:37.210412 5370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:37.210697 5371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb9859656fc46a8b511c65ceef2dbd3. Latest consensus state: current_term: 1 leader_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb9859656fc46a8b511c65ceef2dbd3" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36993 } } }
I20260927 16:32:37.210790 5371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3 [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:37.211215 5378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:37.213739 5378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:37.219189 5378 catalog_manager.cc:1347] Generated new cluster ID: a2cf73c0fb9241179908fb8b02957aca
I20260927 16:32:37.219264 5378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:37.229768 5378 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:37.231002 5378 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:37.238644 5378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb9859656fc46a8b511c65ceef2dbd3: Generated new TSK 0
I20260927 16:32:37.239358 5378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:37.247015 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5291
2026-09-27T16:32:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:37 dist-test-slave-grzf krb5kdc[5390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:37 dist-test-slave-grzf krb5kdc[5390](info): set up 2 sockets
Sep 27 16:32:37 dist-test-slave-grzf krb5kdc[5390](info): commencing operation
krb5kdc: starting...
W20260927 16:32:38.279759 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:38 dist-test-slave-grzf krb5kdc[5390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:38.398675 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36477
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:38.506349 5403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:38.506619 5403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:38.506698 5403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:38.510710 5403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:38.510774 5403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:38.510793 5403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:38.510834 5403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:38.510869 5403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:38.516119 5403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:38.517168 5403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:38.518210 5403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:38.526477 5408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:38.526477 5409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:38.526532 5411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:38.526652 5403 server_base.cc:1034] running on GCE node
I20260927 16:32:38.527405 5403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:38.528476 5403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:38.529644 5403 hybrid_clock.cc:648] HybridClock initialized: now 1790526758529627 us; error 28 us; skew 500 ppm
Sep 27 16:32:38 dist-test-slave-grzf krb5kdc[5390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526758, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:38.532408 5403 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:38.533226 5403 webserver.cc:460] Webserver started at http://127.15.93.190:44609/ using document root <none> and password file <none>
I20260927 16:32:38.533766 5403 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:38.533825 5403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:38.534044 5403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:38.536082 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e"
format_stamp: "Formatted at 2026-09-27 16:32:38 on dist-test-slave-grzf"
server_key: "9e76ef017b105ab34881566ea6317a03"
server_key_iv: "f4b405ccd080626a9ee13e841ba8ae44"
server_key_version: "encryptionkey@0"
I20260927 16:32:38.536623 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e"
format_stamp: "Formatted at 2026-09-27 16:32:38 on dist-test-slave-grzf"
server_key: "9e76ef017b105ab34881566ea6317a03"
server_key_iv: "f4b405ccd080626a9ee13e841ba8ae44"
server_key_version: "encryptionkey@0"
I20260927 16:32:38.540114 5403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:38.542354 5418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:38.543576 5403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:38.543696 5403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e"
format_stamp: "Formatted at 2026-09-27 16:32:38 on dist-test-slave-grzf"
server_key: "9e76ef017b105ab34881566ea6317a03"
server_key_iv: "f4b405ccd080626a9ee13e841ba8ae44"
server_key_version: "encryptionkey@0"
I20260927 16:32:38.543797 5403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:38.548162 5403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:38.551198 5403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:38.551432 5403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:38.557983 5403 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36477
I20260927 16:32:38.558066 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36477 every 8 connection(s)
I20260927 16:32:38.558917 5403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:38.561659 5471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:38.563913 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5403
I20260927 16:32:38.564049 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:38.564340 15734 external_mini_cluster.cc:1428] Setting key b45cc52b513a709962ab7c448c1b5029
I20260927 16:32:38.568684 5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e: Bootstrap starting.
I20260927 16:32:38.569486 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:38.565618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54367 (local address 127.15.93.190:36477)
0927 16:32:38.566009 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:38.566027 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:38.566071 (+ 44us) server_negotiation.cc:407] Connection header received
0927 16:32:38.566804 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:38.566827 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:38.567189 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:38.567574 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:38.568202 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:38.568208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:38.568222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:38.568232 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:38.568285 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:38.568308 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:38.568445 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:38.568510 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 16:32:38.568651 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260927 16:32:38.571190 5471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:38.572005 5471 log.cc:824] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:38.572247 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:38.573989 5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e: No bootstrap required, opened a new log
I20260927 16:32:38.575810 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:38.574297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51711 (local address 127.15.93.190:36477)
0927 16:32:38.574420 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:38.574425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:38.574600 (+ 175us) server_negotiation.cc:407] Connection header received
0927 16:32:38.574693 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:38.574696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:38.574728 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:38.574812 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:38.575481 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:38.575485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:38.575487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 16:32:38.575499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:38.575526 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 16:32:38.575539 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:38.575631 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:38.575664 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 16:32:38.575698 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 16:32:38.577780 5471 raft_consensus.cc:348] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } }
I20260927 16:32:38.577983 5471 raft_consensus.cc:374] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 16:32:38.578054 5471 raft_consensus.cc:729] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006fe09923c4ac69a0a8bd3f5c10e4e, State: Initialized, Role: FOLLOWER
I20260927 16:32:38.578451 5471 consensus_queue.cc:260] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } }
I20260927 16:32:38.578563 5471 raft_consensus.cc:388] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 16:32:38.578644 5471 raft_consensus.cc:482] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 16:32:38.578745 5471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:38.579656 5471 raft_consensus.cc:504] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } }
I20260927 16:32:38.579967 5471 leader_election.cc:302] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f006fe09923c4ac69a0a8bd3f5c10e4e; no voters:
I20260927 16:32:38.580262 5471 leader_election.cc:288] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 16:32:38.580325 5481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:38.580515 5481 raft_consensus.cc:686] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [term 1 LEADER]: Becoming Leader. State: Replica: f006fe09923c4ac69a0a8bd3f5c10e4e, State: Running, Role: LEADER
I20260927 16:32:38.580816 5481 consensus_queue.cc:237] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } }
I20260927 16:32:38.581498 5471 sys_catalog.cc:564] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:38.582468 5482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } } }
I20260927 16:32:38.582700 5483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f006fe09923c4ac69a0a8bd3f5c10e4e. Latest consensus state: current_term: 1 leader_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f006fe09923c4ac69a0a8bd3f5c10e4e" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36477 } } }
I20260927 16:32:38.582803 5482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:38.582906 5483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:38.583789 5491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:38.586673 5491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:38.592005 5491 catalog_manager.cc:1347] Generated new cluster ID: 500c6d66b9fb4c18b28071494a04d4d6
I20260927 16:32:38.592072 5491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:38.603953 5491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:38.604254 5491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:38.611198 5491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f006fe09923c4ac69a0a8bd3f5c10e4e: Generated new TSK 0
I20260927 16:32:38.611966 5491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:38.617936 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5403
2026-09-27T16:32:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (25142 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:38 dist-test-slave-grzf krb5kdc[5502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:38 dist-test-slave-grzf krb5kdc[5502](info): set up 2 sockets
Sep 27 16:32:38 dist-test-slave-grzf krb5kdc[5502](info): commencing operation
krb5kdc: starting...
W20260927 16:32:39.659694 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:39 dist-test-slave-grzf krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:39.775236 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41409
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:39.882755 5515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:39.883030 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:39.883054 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:39.883092 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:39.887001 5515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:39.887058 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:39.887076 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:39.887092 5515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:39.887107 5515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:39.892171 5515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:39.893107 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:39.894032 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:39.902216 5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:39.902226 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:39.902256 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:39.902428 5515 server_base.cc:1034] running on GCE node
I20260927 16:32:39.903074 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:39.904210 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:39.905397 5515 hybrid_clock.cc:648] HybridClock initialized: now 1790526759905368 us; error 42 us; skew 500 ppm
Sep 27 16:32:39 dist-test-slave-grzf krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:39.908107 5515 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:39.908991 5515 webserver.cc:460] Webserver started at http://127.15.93.190:41219/ using document root <none> and password file <none>
I20260927 16:32:39.909528 5515 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:39.909639 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:39.909861 5515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:39.911927 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "561a23a3fbcf4d96b0f84d408a34a00c"
format_stamp: "Formatted at 2026-09-27 16:32:39 on dist-test-slave-grzf"
server_key: "2a4cf740a0f3639f3eb92aa5a86550b6"
server_key_iv: "868ab85fd68348100789e1c806aa3bc2"
server_key_version: "encryptionkey@0"
I20260927 16:32:39.912426 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "561a23a3fbcf4d96b0f84d408a34a00c"
format_stamp: "Formatted at 2026-09-27 16:32:39 on dist-test-slave-grzf"
server_key: "2a4cf740a0f3639f3eb92aa5a86550b6"
server_key_iv: "868ab85fd68348100789e1c806aa3bc2"
server_key_version: "encryptionkey@0"
I20260927 16:32:39.915956 5515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:32:39.918139 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:39.919283 5515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:39.919422 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "561a23a3fbcf4d96b0f84d408a34a00c"
format_stamp: "Formatted at 2026-09-27 16:32:39 on dist-test-slave-grzf"
server_key: "2a4cf740a0f3639f3eb92aa5a86550b6"
server_key_iv: "868ab85fd68348100789e1c806aa3bc2"
server_key_version: "encryptionkey@0"
I20260927 16:32:39.919515 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:39.937183 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:39.939918 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:39.940101 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:39.947153 5515 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:41409
I20260927 16:32:39.947463 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:41409 every 8 connection(s)
I20260927 16:32:39.948101 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:39.950635 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5515
I20260927 16:32:39.950670 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:39.950771 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:39.951028 15734 external_mini_cluster.cc:1428] Setting key 0066dd6a8ad949b51493008f824f7a9c
I20260927 16:32:39.954324 5583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.93.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.189@KRBTEST.COM" created.
Sep 27 16:32:39 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:39.969043 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:39.956581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41409 (local address 127.0.0.1:54454)
0927 16:32:39.957129 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:39.957163 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:39.957942 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:39.958608 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:39.958613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:39.959012 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:39.959296 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:39.959310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:39.960521 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:39.960525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:39.961372 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:39.961382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:39.961993 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:39.961996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:39.962194 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:39.962770 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:39.962785 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:39.964221 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:39.966125 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:39.966133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:39.966143 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:39.966363 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:39.966678 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:39.966680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:39.966682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:39.966790 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:39.967352 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:39.967358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:39.967518 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 16:32:39.967681 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:32:39.968059 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":135,"thread_start_us":145,"threads_started":1}
I20260927 16:32:39.969043 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:39.956580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.15.93.190:41409)
0927 16:32:39.956981 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:39.956998 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:39.957266 (+ 268us) server_negotiation.cc:407] Connection header received
0927 16:32:39.958231 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:39.958235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:39.958439 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:39.958619 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:39.959475 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:39.960395 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:39.961486 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:39.961847 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:39.962059 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:39.964360 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:39.964367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:39.964380 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:39.964407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:39.966005 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:39.966466 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:39.966473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:39.966479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:39.966583 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:39.966876 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:39.966881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:39.966883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:39.967239 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:39.967370 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:39.967753 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 16:32:39.968011 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":262,"thread_start_us":139,"threads_started":1}
I20260927 16:32:39.971473 5583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:39.972173 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:39.971906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53766)
0927 16:32:39.972016 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:39.972093 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:32:39.972393 5533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
I20260927 16:32:39.973250 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:39.973020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53774)
0927 16:32:39.973139 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:39.973190 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 16:32:39.973392 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:39377: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
I20260927 16:32:39.974380 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:39377
--webserver_interface=127.15.93.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:32:40.022620 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } attempt: 1
I20260927 16:32:40.023661 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.023281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53782)
0927 16:32:40.023471 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.023568 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 16:32:40.024382 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.024161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53790)
0927 16:32:40.024264 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.024316 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:32:40.024524 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:39377: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
W20260927 16:32:40.082857 5590 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:40.083122 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:40.083146 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:40.083232 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:40.087116 5590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:40.087169 5590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:40.087188 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:40.087229 5590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:40.087260 5590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:40.092377 5590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:39377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.93.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:40.092728 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } attempt: 2
I20260927 16:32:40.093518 5590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:40.093559 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.093220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53792)
0927 16:32:40.093386 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.093463 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 16:32:40.094321 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.094101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53796)
0927 16:32:40.094207 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.094259 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:32:40.094499 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:39377: Network error: Client connection negotiation failed: client connection to 127.15.93.189:39377: connect: Connection refused (error 111)
I20260927 16:32:40.094739 5590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:40.102864 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:40.102874 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:40.102857 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:40.103061 5590 server_base.cc:1034] running on GCE node
I20260927 16:32:40.103690 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:40.104712 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:40.105885 5590 hybrid_clock.cc:648] HybridClock initialized: now 1790526760105875 us; error 31 us; skew 500 ppm
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:40.108561 5590 init.cc:377] Logged in from keytab as kudu/127.15.93.189@KRBTEST.COM (short username kudu)
I20260927 16:32:40.109361 5590 webserver.cc:460] Webserver started at http://127.15.93.189:39349/ using document root <none> and password file <none>
I20260927 16:32:40.109886 5590 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:40.109968 5590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:40.110194 5590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:40.112179 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data/instance:
uuid: "c70b21af29a54500a93649c29d7194a4"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "4445ba2268589c8ce4ea3f130fb934ed"
server_key_iv: "d5cd464b2a16378aae0f782e9d7520be"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.112707 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal/instance:
uuid: "c70b21af29a54500a93649c29d7194a4"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "4445ba2268589c8ce4ea3f130fb934ed"
server_key_iv: "d5cd464b2a16378aae0f782e9d7520be"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.116415 5590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 16:32:40.118769 5605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:40.120000 5590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 16:32:40.120142 5590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "c70b21af29a54500a93649c29d7194a4"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "4445ba2268589c8ce4ea3f130fb934ed"
server_key_iv: "d5cd464b2a16378aae0f782e9d7520be"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.120249 5590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:40.145582 5590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:40.149325 5590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:40.149533 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:40.156193 5590 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.189:39377
I20260927 16:32:40.156478 5657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.189:39377 every 8 connection(s)
I20260927 16:32:40.157163 5590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
I20260927 16:32:40.159847 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:40.160223 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5590
I20260927 16:32:40.160341 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-1/wal/instance
I20260927 16:32:40.160645 15734 external_mini_cluster.cc:1428] Setting key 6e6f90084272b6a6cec0153925931ec7
I20260927 16:32:40.164544 5658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.93.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.188@KRBTEST.COM" created.
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:40.178557 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.166636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.15.93.190:41409)
0927 16:32:40.166798 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.166804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.167583 (+ 779us) server_negotiation.cc:407] Connection header received
0927 16:32:40.168482 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.168488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.168553 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.168662 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.169758 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.170326 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.171478 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.171838 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.171956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.174419 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.174422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.174424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.174447 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.176580 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.177067 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.177071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.177077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.177148 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.177472 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.177477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.177478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.177655 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.177757 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.178249 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.178374 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:32:40.179296 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.166963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41409 (local address 127.0.0.1:54462)
0927 16:32:40.167425 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.167476 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.168282 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.168800 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.168806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.169176 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.169626 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.169637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.170443 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.170446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.171329 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.171342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.172009 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.172016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.172269 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.172904 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.172919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.174243 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.176692 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.176697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.176712 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.176961 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.177236 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.177240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.177242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.177382 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.177754 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.177760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.178000 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.178181 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.178488 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260927 16:32:40.180757 5658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.185657 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:40.186909 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:43319
--webserver_interface=127.15.93.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:32:40.190812 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.181254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53806)
0927 16:32:40.181359 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.181371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.181431 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.181980 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.181982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.182152 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.182249 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.182254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.183479 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.183481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.184033 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.184042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.184746 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.184749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.184862 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.185278 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.185289 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.186642 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.189006 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.189011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.189012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.189272 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.189597 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.189603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.189608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.189708 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.190201 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.190207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.190316 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.190432 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.190574 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":25}
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:40.191912 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.181363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.15.93.189:39377)
0927 16:32:40.181669 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.181678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.181697 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:32:40.181735 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.181738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.181875 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.181983 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.182337 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.183303 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.184151 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.184631 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.184741 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.186765 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.186776 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.186789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.186815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.188897 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.189397 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.189402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.189404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.189490 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.189801 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.189804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.189806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.190093 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.190216 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.191173 (+ 957us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.191335 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":224,"thread_start_us":132,"threads_started":1}
I20260927 16:32:40.194120 5658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.195027 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.194648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:60970)
0927 16:32:40.194840 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.194945 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 16:32:40.195266 5609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
I20260927 16:32:40.195421 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.186077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53812)
0927 16:32:40.186254 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.186268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.186343 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.187144 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.187149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.187301 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.187442 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.187450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.188819 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.188825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.189507 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.189518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.190070 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.190074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.190199 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.190633 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.190655 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.191957 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.193944 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.193949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.193951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.194197 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.194489 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.194492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.194493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.194582 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.194927 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.194930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.195029 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.195144 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.195270 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 16:32:40.195895 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.186567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.15.93.189:39377)
0927 16:32:40.186898 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.186906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.186918 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:32:40.186952 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.186956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.186998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.187125 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.187852 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.188669 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.189641 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.189931 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.190414 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.192150 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.192154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.192156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.192189 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.193842 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.194309 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.194312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.194314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.194387 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.194675 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.194678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.194680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.194837 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.194936 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.195632 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.195740 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260927 16:32:40.196437 5583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.196684 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.196379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:60986)
0927 16:32:40.196551 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.196608 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 16:32:40.196921 5658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
I20260927 16:32:40.197086 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.196863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:60990)
0927 16:32:40.196950 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.196996 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 16:32:40.197660 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.197440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32770)
0927 16:32:40.197571 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.197618 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 16:32:40.197820 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
I20260927 16:32:40.229005 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 1
I20260927 16:32:40.229833 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.229548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32778)
0927 16:32:40.229690 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.229758 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 16:32:40.230576 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.230362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32794)
0927 16:32:40.230469 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.230520 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:32:40.230728 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
I20260927 16:32:40.246206 5658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 1
I20260927 16:32:40.247092 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.246756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32810)
0927 16:32:40.246925 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.247000 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 16:32:40.247850 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.247624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32812)
0927 16:32:40.247730 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.247785 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 16:32:40.247985 5658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
W20260927 16:32:40.300582 5665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:40.300840 5665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:40.300864 5665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:40.300902 5665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:40.304903 5665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:40.304955 5665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:40.304975 5665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:40.304991 5665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:40.305007 5665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:40.305881 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 2
I20260927 16:32:40.306680 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.306388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32826)
0927 16:32:40.306538 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.306603 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 16:32:40.307483 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.307235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32838)
0927 16:32:40.307327 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.307413 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 16:32:40.307693 5583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
I20260927 16:32:40.310541 5665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:41409,127.15.93.189:39377,127.15.93.188:43319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:43319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.93.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:40.311647 5665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:40.312673 5665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:32:40.316141 5658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 2
I20260927 16:32:40.316924 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.316601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32850)
0927 16:32:40.316760 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.316831 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 16:32:40.317668 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.317478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32866)
0927 16:32:40.317574 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.317619 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 16:32:40.317819 5658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:43319: Network error: Client connection negotiation failed: client connection to 127.15.93.188:43319: connect: Connection refused (error 111)
W20260927 16:32:40.321319 5672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:40.321381 5674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:40.321326 5671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:40.321727 5665 server_base.cc:1034] running on GCE node
I20260927 16:32:40.322227 5665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:40.323351 5665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:40.324563 5665 hybrid_clock.cc:648] HybridClock initialized: now 1790526760324544 us; error 43 us; skew 500 ppm
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:40.327306 5665 init.cc:377] Logged in from keytab as kudu/127.15.93.188@KRBTEST.COM (short username kudu)
I20260927 16:32:40.328229 5665 webserver.cc:460] Webserver started at http://127.15.93.188:41343/ using document root <none> and password file <none>
I20260927 16:32:40.328773 5665 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:40.328830 5665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:40.329048 5665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:40.331022 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data/instance:
uuid: "ec8ed199711146849c5c4248269118e0"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "0cd2b1f7ee93f82e8e1b4bb01e73d053"
server_key_iv: "149eb5ca0c54db2d6254278ba0a23cbe"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.331609 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal/instance:
uuid: "ec8ed199711146849c5c4248269118e0"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "0cd2b1f7ee93f82e8e1b4bb01e73d053"
server_key_iv: "149eb5ca0c54db2d6254278ba0a23cbe"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.335121 5665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 16:32:40.337553 5681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:40.338711 5665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 16:32:40.338828 5665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "ec8ed199711146849c5c4248269118e0"
format_stamp: "Formatted at 2026-09-27 16:32:40 on dist-test-slave-grzf"
server_key: "0cd2b1f7ee93f82e8e1b4bb01e73d053"
server_key_iv: "149eb5ca0c54db2d6254278ba0a23cbe"
server_key_version: "encryptionkey@0"
I20260927 16:32:40.338935 5665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:40.372742 5665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:40.376060 5665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:40.376269 5665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:40.382977 5665 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.188:43319
I20260927 16:32:40.383249 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.188:43319 every 8 connection(s)
I20260927 16:32:40.383934 5665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
I20260927 16:32:40.386615 5734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:40.390375 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.393744 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5665
I20260927 16:32:40.393878 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790526488131879-15734-0/minicluster-data/master-2/wal/instance
I20260927 16:32:40.394145 15734 external_mini_cluster.cc:1428] Setting key 26f89bddc4b9d204a431619a3459fa79
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:40.404136 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.392353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.15.93.190:41409)
0927 16:32:40.392485 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.392489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.393388 (+ 899us) server_negotiation.cc:407] Connection header received
0927 16:32:40.394434 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.394439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.394499 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.394587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.395891 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.396389 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.397473 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.397713 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.397825 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.400560 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.400565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.400570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.400599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.402076 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.402588 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.402593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.402595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.402679 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.403001 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.403004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.403006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.403190 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.403300 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.403881 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.404011 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:40.405103 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.392722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:41409 (local address 127.0.0.1:54472)
0927 16:32:40.393224 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.393268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.394191 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.394789 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.394798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.395226 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.395751 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.395764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.396517 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.396520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.397352 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.397360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.397863 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.397867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.398035 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.398600 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.398616 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.400404 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.402183 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.402193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.402202 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.402489 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.402766 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.402771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.402772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.402903 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.403312 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.403318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.403627 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.403820 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.404118 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":32,"thread_start_us":118,"threads_started":1}
I20260927 16:32:40.405129 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.394829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.15.93.188:43319)
0927 16:32:40.395094 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.395107 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.395304 (+ 197us) server_negotiation.cc:407] Connection header received
0927 16:32:40.395466 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.395471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.395628 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.395781 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.396482 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.397364 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.398188 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.398706 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.398829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.401660 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.401665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.401674 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.401694 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.403177 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.403613 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.403618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.403620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.403698 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.403998 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.404001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.404002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.404254 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.404440 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.404791 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.404920 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260927 16:32:40.406524 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.406868 5583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 3
I20260927 16:32:40.408972 5658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } attempt: 3
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:40.417397 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.407027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:39377 (local address 127.0.0.1:53822)
0927 16:32:40.407274 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.407287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.407356 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.407994 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.407999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.408161 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.408252 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.408263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.409216 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.409222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.409965 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.409974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.410431 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.410535 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.410950 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.410960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.412838 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.415824 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.415828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.415830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.416017 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.416313 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.416318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.416401 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.416882 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.416885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.416972 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.417086 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.417213 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 16:32:40.417766 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.407212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32884)
0927 16:32:40.407499 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.407512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.407588 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.407840 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.407843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.408017 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.408235 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.408244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.409108 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.409111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.409865 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.409874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.410472 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.410553 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.411118 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.411134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.413217 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.415715 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.415720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.415723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.416378 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.416678 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.416684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.416777 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.417236 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.417240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.417345 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.417433 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.417561 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":57,"spinlock_wait_cycles":3328}
I20260927 16:32:40.418406 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.407433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.15.93.188:43319)
0927 16:32:40.407531 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.407535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.407550 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:32:40.407668 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.407674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.407718 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.408126 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.408344 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.408989 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.409998 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.410351 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410751 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.413348 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.413352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.413354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.413375 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.415591 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416477 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.416480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.416482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.416566 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416897 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.416899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.416901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.417124 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.418084 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.418158 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.418269 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:32:40.418883 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.407299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.15.93.189:39377)
0927 16:32:40.407536 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.407543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.407559 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:32:40.407730 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.407734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.407793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.407850 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.408417 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.409086 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410080 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.410330 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.412976 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.412982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.412988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.413020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.415721 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416119 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.416125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.416128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.416213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.416531 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.416537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.416542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.416744 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.416835 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.418577 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.418695 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 16:32:40.419011 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.407648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.15.93.190:41409)
0927 16:32:40.407743 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.407746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.407807 (+ 61us) server_negotiation.cc:407] Connection header received
0927 16:32:40.407912 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.407915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.407955 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.408036 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.408720 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.409415 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.410461 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.411242 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.411322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.414208 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.414213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.414217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.414245 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.416977 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.417439 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.417442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.417444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.417668 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.417972 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.417977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.417980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.418190 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.418403 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.418727 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.418835 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":8064}
I20260927 16:32:40.419731 5734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:40.421134 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.409325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32890)
0927 16:32:40.409544 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.409557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.409624 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.412248 (+ 2624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.412254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.412450 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.412563 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.412571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.413561 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.413567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.414546 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.414558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.415218 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.415225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.415322 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.415788 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.415804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.417109 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.419567 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.419571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.419573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.419835 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.420172 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.420178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.420183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.420280 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.420704 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.420711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.420807 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.420917 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.421034 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 16:32:40.421267 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.409735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.15.93.188:43319)
0927 16:32:40.411959 (+ 2224us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.411968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.411984 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:32:40.412020 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.412023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.412067 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.412200 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.412690 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.413424 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.414699 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.415061 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.415204 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.417237 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.417240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.417243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.417263 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.419454 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.419961 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.419963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.419964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.420072 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.420400 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.420404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.420405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.420582 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.420702 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.420983 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.421112 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2151,"thread_start_us":96,"threads_started":1}
I20260927 16:32:40.422333 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: Bootstrap starting.
I20260927 16:32:40.424479 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: Bootstrap starting.
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:40.425595 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:40.426546 5583 log.cc:824] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:40.426667 5658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:40.427479 5658 log.cc:824] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:40.428822 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.420180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:43319 (local address 127.0.0.1:32904)
0927 16:32:40.420260 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:40.420274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:40.420337 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:40.420599 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:40.420602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:40.420765 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:40.420839 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.420846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.422175 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.422178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.422923 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:40.422934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.423608 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.423611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:40.423693 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.424156 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:40.424170 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:40.425574 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:40.427351 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.427356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.427359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.427576 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.427864 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:40.427867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:40.427869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:40.427951 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:40.428373 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:40.428376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:40.428468 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 16:32:40.428557 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 16:32:40.428676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 16:32:40.428953 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: No bootstrap required, opened a new log
I20260927 16:32:40.429364 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.420348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.15.93.188:43319)
0927 16:32:40.420433 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.420437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.420446 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:32:40.420477 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.420479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.420512 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.420589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.421411 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.422051 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.423039 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.423352 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.423476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.425710 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.425713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.425716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.425739 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.427249 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.427675 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.427679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.427680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.427764 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.428046 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.428052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.428057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.428268 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.429077 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.429142 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.429238 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:32:40.430972 5658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: No bootstrap required, opened a new log
I20260927 16:32:40.431938 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: Bootstrap starting.
I20260927 16:32:40.432936 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.433147 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561a23a3fbcf4d96b0f84d408a34a00c, State: Initialized, Role: FOLLOWER
I20260927 16:32:40.433449 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.433921 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.434058 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.434248 5734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:40.434695 5658 raft_consensus.cc:348] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.434788 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:40.434835 5658 raft_consensus.cc:729] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70b21af29a54500a93649c29d7194a4, State: Initialized, Role: FOLLOWER
I20260927 16:32:40.434990 5734 log.cc:824] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:40.435043 5658 consensus_queue.cc:260] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.435408 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.435540 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 16:32:40 dist-test-slave-grzf krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:40.436291 5658 sys_catalog.cc:564] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 16:32:40.439764 5755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:40.439891 5755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:40.441213 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.424219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.15.93.189:39377)
0927 16:32:40.427924 (+ 3705us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.427928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.427943 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:32:40.427969 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.427971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.428022 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.428874 (+ 852us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.428964 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.429678 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.430704 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.431027 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.432824 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.436944 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.436950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.436953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.436984 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.438780 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.439301 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.439305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.439306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.439435 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.439848 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.439855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.439860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.440092 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.440322 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.440964 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.441075 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3641}
I20260927 16:32:40.441536 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: No bootstrap required, opened a new log
W20260927 16:32:40.441656 5536 tablet.cc:2369] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:40.445578 5734 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.445806 5734 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8ed199711146849c5c4248269118e0, State: Initialized, Role: FOLLOWER
I20260927 16:32:40.446126 5734 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
W20260927 16:32:40.446228 5767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:40.446285 5767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:40.443943 5515 master_runner.cc:186] Error getting master registration for 127.15.93.188:43319: OK, instance_id { permanent_uuid: "ec8ed199711146849c5c4248269118e0" instance_seqno: 1790526760381127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:32:40.446652 5515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:32:40.447628 5734 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:40.449869 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:40.451177 5557 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "561a23a3fbcf4d96b0f84d408a34a00c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474
I20260927 16:32:40.451457 5557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:32:40.451581 5557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:40.452889 5557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.453822 5557 leader_election.cc:288] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [CANDIDATE]: Term 1 election: Requested vote from peers c70b21af29a54500a93649c29d7194a4 (127.15.93.189:39377), ec8ed199711146849c5c4248269118e0 (127.15.93.188:43319)
I20260927 16:32:40.451150 5590 master_runner.cc:186] Error getting master registration for 127.15.93.188:43319: OK, instance_id { permanent_uuid: "ec8ed199711146849c5c4248269118e0" instance_seqno: 1790526760381127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:32:40.454408 5632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c70b21af29a54500a93649c29d7194a4"
I20260927 16:32:40.454586 5590 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:32:40.454663 5632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:40.455235 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec8ed199711146849c5c4248269118e0"
I20260927 16:32:40.455466 5708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:40.456054 5632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561a23a3fbcf4d96b0f84d408a34a00c in term 1.
I20260927 16:32:40.456513 5533 leader_election.cc:302] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561a23a3fbcf4d96b0f84d408a34a00c, c70b21af29a54500a93649c29d7194a4; no voters:
I20260927 16:32:40.456821 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:40.457013 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [term 1 LEADER]: Becoming Leader. State: Replica: 561a23a3fbcf4d96b0f84d408a34a00c, State: Running, Role: LEADER
I20260927 16:32:40.457343 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } }
I20260927 16:32:40.457413 5708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561a23a3fbcf4d96b0f84d408a34a00c in term 1.
I20260927 16:32:40.459035 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561a23a3fbcf4d96b0f84d408a34a00c. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.459154 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:40.459594 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.459787 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.460150 5787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:40.461159 5787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:40.462922 5665 master_runner.cc:263] No leader master found from master ec8ed199711146849c5c4248269118e0
I20260927 16:32:40.462982 5665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:32:40.467185 5632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [term 1 FOLLOWER]: Refusing update from remote peer 561a23a3fbcf4d96b0f84d408a34a00c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:40.468088 5708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [term 1 FOLLOWER]: Refusing update from remote peer 561a23a3fbcf4d96b0f84d408a34a00c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:40.468820 5785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:40.469913 5785 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 16:32:40.470512 5780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:40.470587 5780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:40.473788 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561a23a3fbcf4d96b0f84d408a34a00c. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.473896 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.475800 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.464815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.15.93.188:43319)
0927 16:32:40.464911 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.464915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.466926 (+ 2011us) server_negotiation.cc:407] Connection header received
0927 16:32:40.467095 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.467098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.467141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.467189 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.468123 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.468851 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.469962 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.470337 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.470410 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.472214 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.472218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.472220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.472244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.473927 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.474450 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.474454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.474455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.474538 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.474854 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.474858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.474859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.475063 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.475173 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.475547 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.475661 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 16:32:40.477654 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561a23a3fbcf4d96b0f84d408a34a00c. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.477792 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.482802 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.482916 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:40.483430 5788 mvcc.cc:205] Tried to move back new op lower bound from 7333997610864807936 to 7333997610836840448. Current Snapshot: MvccSnapshot[applied={T|T < 7333997610864807936}]
I20260927 16:32:40.484786 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.484886 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.485419 5793 mvcc.cc:205] Tried to move back new op lower bound from 7333997610864807936 to 7333997610836840448. Current Snapshot: MvccSnapshot[applied={T|T < 7333997610864807936}]
I20260927 16:32:40.492640 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.492763 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:40.494232 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "561a23a3fbcf4d96b0f84d408a34a00c" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 41409 } } peers { permanent_uuid: "c70b21af29a54500a93649c29d7194a4" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 39377 } } peers { permanent_uuid: "ec8ed199711146849c5c4248269118e0" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 43319 } } }
I20260927 16:32:40.494364 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:40.494837 5787 catalog_manager.cc:1347] Generated new cluster ID: 89a9931eaa344f3caac742c36ed40d09
I20260927 16:32:40.494899 5787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:40.499722 5795 mvcc.cc:205] Tried to move back new op lower bound from 7333997610864807936 to 7333997610836840448. Current Snapshot: MvccSnapshot[applied={T|T < 7333997610864807936}]
I20260927 16:32:40.507674 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.463829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.15.93.189:39377)
0927 16:32:40.463939 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.463943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.467412 (+ 3469us) server_negotiation.cc:407] Connection header received
0927 16:32:40.467454 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.467458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.467508 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.467601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.478230 (+ 10629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.479002 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.485642 (+ 6640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.485989 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.486067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.490284 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.490288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.490291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.490321 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.492064 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.503971 (+ 11907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.503974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.503976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.504069 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.504372 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.504375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.504377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.504591 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.504703 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.507324 (+ 2621us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.507495 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:32:40.511039 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:40.464209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.15.93.190:41409)
0927 16:32:40.464305 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.464309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.465609 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 16:32:40.465805 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.465811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.465862 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.465918 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.479137 (+ 13219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.479879 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.487683 (+ 7804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:40.488029 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:40.488107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:40.504804 (+ 16697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:40.504808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:40.504811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:40.504839 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:40.506534 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.509465 (+ 2931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.509469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.509471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.509582 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:40.509963 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:40.509966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:40.509968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:40.510189 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:40.510303 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:40.510734 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 16:32:40.510870 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:40.524526 5787 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:40.525861 5787 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:40.542194 5787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 561a23a3fbcf4d96b0f84d408a34a00c: Generated new TSK 0
I20260927 16:32:40.543141 5787 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 16:32:40.568894 5586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.565782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.15.93.190:41409)
0927 16:32:40.565904 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.565909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.567653 (+ 1744us) server_negotiation.cc:407] Connection header received
0927 16:32:40.567839 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.567843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.567895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.567949 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.568757 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54496: BlockingRecv error: recv got EOF from 127.0.0.1:54496 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 16:32:40.570307 5737 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.566469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.15.93.188:43319)
0927 16:32:40.566574 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.566579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.568206 (+ 1627us) server_negotiation.cc:407] Connection header received
0927 16:32:40.569593 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.569596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.569644 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.569695 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.570192 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32914: BlockingRecv error: recv got EOF from 127.0.0.1:32914 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 16:32:40.571632 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.567034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.15.93.189:39377)
0927 16:32:40.567143 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.567147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.569404 (+ 2257us) server_negotiation.cc:407] Connection header received
0927 16:32:40.570893 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.570896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.570946 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.570997 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.571512 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53854: BlockingRecv error: recv got EOF from 127.0.0.1:53854 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:32:40.576150 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.574682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.15.93.189:39377)
0927 16:32:40.574782 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.574786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.575227 (+ 441us) server_negotiation.cc:407] Connection header received
0927 16:32:40.575459 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.575462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.575536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.575599 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.576054 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53864: BlockingRecv error: recv got EOF from 127.0.0.1:53864 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 16:32:40.580860 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.579479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.15.93.189:39377)
0927 16:32:40.579579 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.579582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.580054 (+ 472us) server_negotiation.cc:407] Connection header received
0927 16:32:40.580221 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.580224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.580267 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.580314 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.580775 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53876: BlockingRecv error: recv got EOF from 127.0.0.1:53876 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 16:32:40.586066 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.584753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.15.93.189:39377)
0927 16:32:40.584839 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.584843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.585299 (+ 456us) server_negotiation.cc:407] Connection header received
0927 16:32:40.585458 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.585461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.585502 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.585547 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.585983 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53882: BlockingRecv error: recv got EOF from 127.0.0.1:53882 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 16:32:40.593186 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.591849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.15.93.189:39377)
0927 16:32:40.591957 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.591960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.592419 (+ 459us) server_negotiation.cc:407] Connection header received
0927 16:32:40.592592 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.592595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.592636 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.592687 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.593108 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53894: BlockingRecv error: recv got EOF from 127.0.0.1:53894 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 16:32:40.604506 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.603081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.15.93.189:39377)
0927 16:32:40.603177 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.603182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.603700 (+ 518us) server_negotiation.cc:407] Connection header received
0927 16:32:40.603867 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.603870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.603912 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.603959 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.604414 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 16:32:40.625237 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.623075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.15.93.189:39377)
0927 16:32:40.623205 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.623209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.624043 (+ 834us) server_negotiation.cc:407] Connection header received
0927 16:32:40.624319 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.624322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.624378 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.624434 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.625122 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53918: BlockingRecv error: recv got EOF from 127.0.0.1:53918 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 16:32:40.661940 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.659875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.15.93.189:39377)
0927 16:32:40.659994 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.659999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.660731 (+ 732us) server_negotiation.cc:407] Connection header received
0927 16:32:40.660938 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.660942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.661104 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.661182 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.661825 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53922: BlockingRecv error: recv got EOF from 127.0.0.1:53922 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 16:32:40.729632 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.728536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.15.93.189:39377)
0927 16:32:40.728722 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.728726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.728831 (+ 105us) server_negotiation.cc:407] Connection header received
0927 16:32:40.728955 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.728958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.729006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.729083 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.729482 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53936: BlockingRecv error: recv got EOF from 127.0.0.1:53936 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 16:32:40.860674 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:40.859585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.15.93.189:39377)
0927 16:32:40.859736 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:40.859743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:40.859858 (+ 115us) server_negotiation.cc:407] Connection header received
0927 16:32:40.859997 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:40.860000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:40.860049 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:40.860122 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:40.860550 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53940: BlockingRecv error: recv got EOF from 127.0.0.1:53940 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 16:32:41.119797 5664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:41.118778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.15.93.189:39377)
0927 16:32:41.118929 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.118932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.118949 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:32:41.119088 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.119091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.119137 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.119210 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:41.119654 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53944: BlockingRecv error: recv got EOF from 127.0.0.1:53944 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 16:32:41.447511 5767 catalog_manager.cc:1259] Loaded cluster ID: 89a9931eaa344f3caac742c36ed40d09
I20260927 16:32:41.447590 5767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: loading cluster ID for follower catalog manager: success
I20260927 16:32:41.449407 5767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: acquiring CA information for follower catalog manager: success
I20260927 16:32:41.450527 5767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c70b21af29a54500a93649c29d7194a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:32:41.471799 5780 catalog_manager.cc:1259] Loaded cluster ID: 89a9931eaa344f3caac742c36ed40d09
I20260927 16:32:41.471880 5780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: loading cluster ID for follower catalog manager: success
I20260927 16:32:41.473593 5780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: acquiring CA information for follower catalog manager: success
I20260927 16:32:41.474730 5780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec8ed199711146849c5c4248269118e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:32:41.638128 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:41.633676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.15.93.189:39377)
0927 16:32:41.633924 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.633929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.634032 (+ 103us) server_negotiation.cc:407] Connection header received
0927 16:32:41.634212 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.634216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.634409 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.634527 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:41.634775 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.635474 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.636461 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.636768 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.636876 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:41.637208 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:41.637703 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:41.637831 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:41.637898 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 16:32:41.637957 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260927 16:32:41.644469 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:41.641043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.15.93.188:43319)
0927 16:32:41.641271 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.641274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.641287 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:32:41.641395 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.641397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.641507 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.641610 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:41.641813 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.642343 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.643161 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.643436 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.643512 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:41.643786 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:41.644175 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:41.644261 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:41.644314 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 16:32:41.644357 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260927 16:32:41.651413 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:41.647055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.15.93.189:39377)
0927 16:32:41.647175 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.647179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.647609 (+ 430us) server_negotiation.cc:407] Connection header received
0927 16:32:41.647820 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.647826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.647887 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.647959 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:41.648346 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.649086 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.650367 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.650624 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.650698 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:41.650944 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:41.651017 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:41.651089 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:41.651198 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 16:32:41.651251 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 16:32:41.651708 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:41.647044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.15.93.190:41409)
0927 16:32:41.647462 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.647467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.647483 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:32:41.647574 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.647579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.647740 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.647877 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:41.648129 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.648899 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.649878 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.650117 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.650226 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:41.650530 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:41.650826 (+ 296us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:41.650923 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:41.651487 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 16:32:41.651553 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":56,"threads_started":1}
I20260927 16:32:41.651902 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:41.647444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.15.93.188:43319)
0927 16:32:41.647541 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:41.647545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:41.647844 (+ 299us) server_negotiation.cc:407] Connection header received
0927 16:32:41.648109 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:41.648112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:41.648159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:41.648205 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:41.648509 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.648995 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.650754 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:41.651001 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:41.651113 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:41.651409 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:41.651484 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:41.651557 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:41.651740 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 16:32:41.651789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:32:41.658105 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5515
I20260927 16:32:41.664597 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5590
I20260927 16:32:41.671082 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5665
2026-09-27T16:32:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3055 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 16:32:41 dist-test-slave-grzf krb5kdc[5883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 16:32:41 dist-test-slave-grzf krb5kdc[5883](info): set up 2 sockets
Sep 27 16:32:41 dist-test-slave-grzf krb5kdc[5883](info): commencing operation
krb5kdc: starting...
W20260927 16:32:42.699808 15734 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/test-user.keytab.
Sep 27 16:32:42 dist-test-slave-grzf krb5kdc[5883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T16:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T16:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.93.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
Entry for principal kudu/127.15.93.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab.
I20260927 16:32:42.816643 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36781
--webserver_interface=127.15.93.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 16:32:42.924515 5896 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:42.924783 5896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:42.924808 5896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:42.924881 5896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:42.928815 5896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:42.928869 5896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:42.928889 5896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:42.928905 5896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:42.928920 5896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:42.934121 5896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.190.keytab
--principal=kudu/127.15.93.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.190:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.93.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:42.935125 5896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:42.936108 5896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:42.944299 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:42.944356 5904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:42.944443 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:42.944707 5896 server_base.cc:1034] running on GCE node
I20260927 16:32:42.945221 5896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:42.946205 5896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:42.947357 5896 hybrid_clock.cc:648] HybridClock initialized: now 1790526762947309 us; error 62 us; skew 500 ppm
Sep 27 16:32:42 dist-test-slave-grzf krb5kdc[5883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:42.950059 5896 init.cc:377] Logged in from keytab as kudu/127.15.93.190@KRBTEST.COM (short username kudu)
I20260927 16:32:42.950912 5896 webserver.cc:460] Webserver started at http://127.15.93.190:41989/ using document root <none> and password file <none>
I20260927 16:32:42.951491 5896 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:42.951555 5896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:42.951790 5896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:42.953826 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data/instance:
uuid: "709fd63acc5c457d9dec73ac31cc943b"
format_stamp: "Formatted at 2026-09-27 16:32:42 on dist-test-slave-grzf"
server_key: "a5ba6cbe4ab0ad56af1e502309beaa62"
server_key_iv: "59e26b50f194627158b319d01fbed881"
server_key_version: "encryptionkey@0"
I20260927 16:32:42.954370 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance:
uuid: "709fd63acc5c457d9dec73ac31cc943b"
format_stamp: "Formatted at 2026-09-27 16:32:42 on dist-test-slave-grzf"
server_key: "a5ba6cbe4ab0ad56af1e502309beaa62"
server_key_iv: "59e26b50f194627158b319d01fbed881"
server_key_version: "encryptionkey@0"
I20260927 16:32:42.958065 5896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 16:32:42.960400 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:42.961632 5896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:42.961767 5896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal
uuid: "709fd63acc5c457d9dec73ac31cc943b"
format_stamp: "Formatted at 2026-09-27 16:32:42 on dist-test-slave-grzf"
server_key: "a5ba6cbe4ab0ad56af1e502309beaa62"
server_key_iv: "59e26b50f194627158b319d01fbed881"
server_key_version: "encryptionkey@0"
I20260927 16:32:42.961869 5896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:42.975468 5896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:42.978152 5896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:42.978351 5896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:42.985076 5896 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.190:36781
I20260927 16:32:42.985333 5963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.190:36781 every 8 connection(s)
I20260927 16:32:42.986019 5896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/data/info.pb
I20260927 16:32:42.988677 5964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:42.992220 5964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:42.992758 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5896
I20260927 16:32:42.992870 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-0/wal/instance
I20260927 16:32:42.993131 15734 external_mini_cluster.cc:1428] Setting key 8f904694609a877c85347a0923948048
WARNING: no policy specified for kudu/127.15.93.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.189@KRBTEST.COM" created.
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:43.009930 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:42.994667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.15.93.190:36781)
0927 16:32:42.995248 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:42.995262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:42.995307 (+ 45us) server_negotiation.cc:407] Connection header received
0927 16:32:42.996352 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:42.996360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:42.996607 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:42.996813 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:42.997640 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:42.999196 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.000197 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.000813 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.000999 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.004995 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.005005 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.005016 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.005036 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.006782 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.007264 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.007270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.007272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.007423 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.007738 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.007741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.007743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.008137 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.008270 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.008631 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.008890 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":457,"spinlock_wait_cycles":50944,"thread_start_us":178,"threads_started":1}
I20260927 16:32:43.009929 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:42.994668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36781 (local address 127.0.0.1:55954)
0927 16:32:42.995159 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:42.995210 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:42.996042 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:42.996824 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:42.996828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:42.997169 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:42.997486 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:42.997500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:42.999311 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:42.999313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.000082 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.000091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.000963 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.000966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.001056 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.001654 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.001670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.004837 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.006908 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.006917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.006928 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.007156 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.007509 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.007515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.007517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.007640 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.008251 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.008258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.008398 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.008568 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.008943 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":108,"thread_start_us":138,"threads_started":1}
I20260927 16:32:43.012840 5964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:43.013650 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.013333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49184)
0927 16:32:43.013446 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.013538 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:32:43.013830 5912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
I20260927 16:32:43.014631 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.014432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49192)
0927 16:32:43.014531 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.014581 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:32:43.014814 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:38159: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
Entry for principal kudu/127.15.93.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab.
I20260927 16:32:43.018958 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:38159
--webserver_interface=127.15.93.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 16:32:43.064059 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } attempt: 1
I20260927 16:32:43.064993 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.064677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49202)
0927 16:32:43.064836 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.064902 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 16:32:43.065791 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.065544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49208)
0927 16:32:43.065662 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.065722 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 16:32:43.065948 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:38159: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
W20260927 16:32:43.129813 5971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:43.130102 5971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:43.130126 5971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:43.130163 5971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 16:32:43.134142 5971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:43.134217 5971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260927 16:32:43.134145 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } attempt: 2
W20260927 16:32:43.134244 5971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:43.134266 5971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:43.134286 5971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:43.135027 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.134661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49222)
0927 16:32:43.134831 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.134906 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 16:32:43.135860 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.135630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49238)
0927 16:32:43.135743 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.135796 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 16:32:43.135998 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.189:38159: Network error: Client connection negotiation failed: client connection to 127.15.93.189:38159: connect: Connection refused (error 111)
I20260927 16:32:43.140012 5971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.189.keytab
--principal=kudu/127.15.93.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.189:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.93.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:43.141187 5971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:43.142166 5971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 16:32:43.150439 5976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:43.150624 5971 server_base.cc:1034] running on GCE node
W20260927 16:32:43.150420 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:43.150420 5977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:43.151365 5971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:43.152504 5971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:43.153664 5971 hybrid_clock.cc:648] HybridClock initialized: now 1790526763153632 us; error 48 us; skew 500 ppm
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:43.156476 5971 init.cc:377] Logged in from keytab as kudu/127.15.93.189@KRBTEST.COM (short username kudu)
I20260927 16:32:43.157369 5971 webserver.cc:460] Webserver started at http://127.15.93.189:42927/ using document root <none> and password file <none>
I20260927 16:32:43.157923 5971 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:43.158008 5971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:43.158234 5971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:43.160300 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data/instance:
uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "824b918cd1e828dfc40bd12e9235c00f"
server_key_iv: "1747c68381869481600ce85f8266c960"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.160849 5971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal/instance:
uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "824b918cd1e828dfc40bd12e9235c00f"
server_key_iv: "1747c68381869481600ce85f8266c960"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.164444 5971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 16:32:43.166787 5986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:43.168031 5971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:43.168165 5971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal
uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "824b918cd1e828dfc40bd12e9235c00f"
server_key_iv: "1747c68381869481600ce85f8266c960"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.168268 5971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:43.190881 5971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:43.193861 5971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:43.194082 5971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:43.201028 5971 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.189:38159
I20260927 16:32:43.201318 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.189:38159 every 8 connection(s)
I20260927 16:32:43.202025 5971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/data/info.pb
I20260927 16:32:43.204756 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 5971
I20260927 16:32:43.204883 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-1/wal/instance
I20260927 16:32:43.204888 6039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:43.205179 15734 external_mini_cluster.cc:1428] Setting key a861bba6fbc202f5ee21fb04b81fea25
I20260927 16:32:43.208635 6039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.93.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.93.188@KRBTEST.COM" created.
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:43.222887 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.210744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.15.93.190:36781)
0927 16:32:43.210916 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.210922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.211726 (+ 804us) server_negotiation.cc:407] Connection header received
0927 16:32:43.212655 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.212661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.212726 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.212852 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.213985 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.214540 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.215863 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.216103 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.216217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.218904 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.218912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.218919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.218956 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.220753 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.221237 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.221244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.221247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.221339 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.221665 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.221670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.221673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.221921 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.222043 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.222510 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.222652 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 16:32:43.223580 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.211087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36781 (local address 127.0.0.1:55970)
0927 16:32:43.211565 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.211617 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.212428 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.212998 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.213012 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.213343 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.213837 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.213851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.214709 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.214711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.215741 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.215751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.216253 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.216258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.216452 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.217087 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.217111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.218690 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.220870 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.220875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.220886 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.221137 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.221433 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.221436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.221437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.221573 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.222021 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.222026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.222256 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.222405 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.222663 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260927 16:32:43.225051 6039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:43.227146 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
Entry for principal kudu/127.15.93.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab.
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:43.230901 15734 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
/tmp/dist-test-task5Uwdq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:36847
--webserver_interface=127.15.93.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.93.148:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:43.235152 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.225532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49248)
0927 16:32:43.225664 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.225676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.225739 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.226276 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.226278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.226443 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.226536 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.226541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.227724 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.227730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.228332 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.228340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.229037 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.229040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.229153 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.229695 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.229706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.231166 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.233188 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.233194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.233197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.233502 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.233841 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.233845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.233848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.233928 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.234520 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.234526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.234638 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.234742 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.234886 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:32:43.235335 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.225613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.15.93.189:38159)
0927 16:32:43.225936 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.225954 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.225973 (+ 19us) server_negotiation.cc:407] Connection header received
0927 16:32:43.226020 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.226024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.226168 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.226272 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.226630 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.227574 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.228440 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.228882 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.229000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.231303 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.231313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.231327 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.231348 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.233087 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.233621 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.233629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.233632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.233738 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.234030 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.234036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.234038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.234386 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.234819 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.234949 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.235102 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
I20260927 16:32:43.237192 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.227608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.15.93.189:38159)
0927 16:32:43.227949 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.227956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.227965 (+ 9us) server_negotiation.cc:407] Connection header received
0927 16:32:43.227989 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.227991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.228020 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.228108 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.228487 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.229082 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.230635 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.230916 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.231008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.233163 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.233167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.233169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.233191 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.235456 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.235919 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.235922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.235924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.236012 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.236234 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.236237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.236239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.236448 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.236549 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.236918 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.237020 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260927 16:32:43.237265 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.227576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49262)
0927 16:32:43.227682 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.227697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.227787 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.228129 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.228132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.228295 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.228383 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.228388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.229193 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.229195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.230508 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.230519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.231022 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.231025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.231135 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.231680 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.231700 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.233030 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.235611 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.235615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.235617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.235835 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.236095 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.236097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.236098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.236161 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.236600 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.236606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.236730 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.236855 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.237027 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 16:32:43.237660 6039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:43.238224 5964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:43.238477 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.238187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33096)
0927 16:32:43.238307 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.238398 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 16:32:43.238767 5989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
I20260927 16:32:43.238911 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.238641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33102)
0927 16:32:43.238799 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.238851 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 16:32:43.239650 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.239396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33114)
0927 16:32:43.239560 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.239606 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260927 16:32:43.239771 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
I20260927 16:32:43.239938 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.239652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33116)
0927 16:32:43.239809 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.239863 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 16:32:43.240096 6039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
I20260927 16:32:43.270911 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 1
I20260927 16:32:43.271802 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.271463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33124)
0927 16:32:43.271610 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.271675 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 16:32:43.272511 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.272243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33138)
0927 16:32:43.272417 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.272458 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260927 16:32:43.272639 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
I20260927 16:32:43.289368 6039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 1
I20260927 16:32:43.290216 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.289890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33144)
0927 16:32:43.290056 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.290130 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 16:32:43.290902 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.290694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33160)
0927 16:32:43.290792 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.290841 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 16:32:43.291049 6039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
W20260927 16:32:43.344225 6046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 16:32:43.344493 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 16:32:43.344570 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 16:32:43.344669 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 16:32:43.347786 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 2
I20260927 16:32:43.348532 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.348260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33172)
0927 16:32:43.348394 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.348457 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 16:32:43.348814 6046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 16:32:43.348881 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 16:32:43.348924 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 16:32:43.348963 6046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 16:32:43.348997 6046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 16:32:43.349205 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.349018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33184)
0927 16:32:43.349106 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.349155 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 16:32:43.349359 5964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
I20260927 16:32:43.354254 6046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.93.148:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.93.190:36781,127.15.93.189:38159,127.15.93.188:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/krb5kdc/kudu_127.15.93.188.keytab
--principal=kudu/127.15.93.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.93.188:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.93.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 15:53:41 UTC on ccycloud.quasar-duzgkl.root.comops.site
build id 4676199
I20260927 16:32:43.355304 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 16:32:43.356303 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 16:32:43.359231 6039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 2
I20260927 16:32:43.359936 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.359698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33200)
0927 16:32:43.359819 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.359872 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 16:32:43.360648 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.360400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33208)
0927 16:32:43.360527 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.360579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 16:32:43.360792 6039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.93.188:36847: Network error: Client connection negotiation failed: client connection to 127.15.93.188:36847: connect: Connection refused (error 111)
W20260927 16:32:43.364562 6052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:43.364539 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 16:32:43.364527 6053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 16:32:43.364800 6046 server_base.cc:1034] running on GCE node
I20260927 16:32:43.365495 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 16:32:43.366605 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 16:32:43.367781 6046 hybrid_clock.cc:648] HybridClock initialized: now 1790526763367762 us; error 32 us; skew 500 ppm
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 16:32:43.370338 6046 init.cc:377] Logged in from keytab as kudu/127.15.93.188@KRBTEST.COM (short username kudu)
I20260927 16:32:43.371183 6046 webserver.cc:460] Webserver started at http://127.15.93.188:32973/ using document root <none> and password file <none>
I20260927 16:32:43.371745 6046 fs_manager.cc:359] Metadata directory not provided
I20260927 16:32:43.371803 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 16:32:43.372000 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 16:32:43.373996 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data/instance:
uuid: "2d79ce82957d431a88f5ea45a85882d8"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "be49b0f8533d432237eb937f5a36d4bd"
server_key_iv: "ef3e2fbc8b02ac49ce68ddd69332f509"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.374496 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal/instance:
uuid: "2d79ce82957d431a88f5ea45a85882d8"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "be49b0f8533d432237eb937f5a36d4bd"
server_key_iv: "ef3e2fbc8b02ac49ce68ddd69332f509"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.377982 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 16:32:43.380219 6062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 16:32:43.381350 6046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 16:32:43.381477 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data,/tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal
uuid: "2d79ce82957d431a88f5ea45a85882d8"
format_stamp: "Formatted at 2026-09-27 16:32:43 on dist-test-slave-grzf"
server_key: "be49b0f8533d432237eb937f5a36d4bd"
server_key_iv: "ef3e2fbc8b02ac49ce68ddd69332f509"
server_key_version: "encryptionkey@0"
I20260927 16:32:43.381577 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 16:32:43.393306 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 16:32:43.395895 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 16:32:43.396085 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 16:32:43.402326 6046 rpc_server.cc:307] RPC server started. Bound to: 127.15.93.188:36847
I20260927 16:32:43.402593 6114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.93.188:36847 every 8 connection(s)
I20260927 16:32:43.403249 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/data/info.pb
I20260927 16:32:43.405818 6115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 16:32:43.408954 15734 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu as pid 6046
I20260927 16:32:43.409076 15734 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Uwdq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790526488131879-15734-0/minicluster-data/master-2/wal/instance
I20260927 16:32:43.409358 15734 external_mini_cluster.cc:1428] Setting key 94639ad2791769081dc1b955701cfe97
I20260927 16:32:43.409591 6115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:43.421636 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.411442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.15.93.190:36781)
0927 16:32:43.411603 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.411607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.411935 (+ 328us) server_negotiation.cc:407] Connection header received
0927 16:32:43.412286 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.412288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.412356 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.412445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.413339 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.413828 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.414896 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.415165 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.415245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.418029 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.418032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.418034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.418058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.419612 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.420179 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.420183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.420185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.420267 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.420585 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.420590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.420592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.420779 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.420882 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.421341 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.421456 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 16:32:43.422454 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.411339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.190:36781 (local address 127.0.0.1:55972)
0927 16:32:43.411785 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.411834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.412100 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.412503 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.412508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.412901 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.413195 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.413208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.413927 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.413931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.414763 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.414772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.415292 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.415296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.415534 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.416311 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.416326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.417868 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.419715 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.419725 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.419735 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.420082 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.420354 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.420359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.420361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.420494 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.420881 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.420888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.421137 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.421300 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.421592 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":44,"spinlock_wait_cycles":1280,"thread_start_us":174,"threads_started":1}
I20260927 16:32:43.422456 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.410670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.15.93.188:36847)
0927 16:32:43.411052 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.411066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.411101 (+ 35us) server_negotiation.cc:407] Connection header received
0927 16:32:43.411943 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.411964 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.412222 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.412385 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.413128 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.414090 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.414994 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.415464 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.415595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.418418 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.418428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.418442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.418468 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.420143 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.420563 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.420568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.420570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.420648 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.420917 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.420921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.420923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.421162 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.421340 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.421578 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.421711 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":243,"spinlock_wait_cycles":3968,"thread_start_us":96,"threads_started":1}
I20260927 16:32:43.423655 6115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.190@KRBTEST.COM
I20260927 16:32:43.432085 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.424118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.189:38159 (local address 127.0.0.1:49278)
0927 16:32:43.424282 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.424296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.424366 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.424640 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.424642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.424810 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.424927 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.424933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.425699 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.425701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.426191 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.426196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.426666 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.426670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.426760 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.427212 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.427225 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.428637 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.430477 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.430479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.430481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.430785 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.431053 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.431057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.431059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.431152 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.431616 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.431619 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.431704 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.431803 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.431928 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 16:32:43.432142 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.424223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.15.93.189:38159)
0927 16:32:43.424362 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.424367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.424382 (+ 15us) server_negotiation.cc:407] Connection header received
0927 16:32:43.424480 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.424485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.424542 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.424629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.425065 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.425585 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.426309 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.426562 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.426669 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.428782 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.428785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.428788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.428818 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.430383 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.430881 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.430884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.430885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.430959 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.431238 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.431240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.431241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.431488 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.431595 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.431864 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.431976 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:32:43.432678 6115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 16:32:43.433122 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.425001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.15.93.190:36781)
0927 16:32:43.425121 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.425126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.425138 (+ 12us) server_negotiation.cc:407] Connection header received
0927 16:32:43.425230 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.425233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.425269 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.425354 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.425915 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.426382 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.427138 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.427356 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.427484 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.429749 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.429753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.429756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.429779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.431336 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.431760 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.431764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.431766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.431849 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.432109 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.432113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.432115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.432349 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.432555 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.432853 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.432967 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.188@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.93.189@KRBTEST.COM
I20260927 16:32:43.441030 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.433140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33218)
0927 16:32:43.433340 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.433353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.433425 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.433671 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.433672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.433846 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.433930 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.433936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.434929 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.434932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.435580 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.435589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.435997 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.436001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.436093 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.436484 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.436495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.437867 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.439568 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.439572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.439573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.439832 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.440121 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.440123 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.440124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.440225 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.440676 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.440678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.440750 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.440831 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.440921 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 16:32:43.441195 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.433275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.15.93.188:36847)
0927 16:32:43.433361 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.433365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.433383 (+ 18us) server_negotiation.cc:407] Connection header received
0927 16:32:43.433519 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.433522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.433563 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.433650 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.434068 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.434817 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.435679 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.435894 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.435969 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.437995 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.437999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.438001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.438032 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.439469 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.439924 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.439927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.439929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.440026 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.440332 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.440337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.440340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.440573 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.440684 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.440910 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.441034 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 16:32:43.442799 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.434323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.15.93.189:38159)
0927 16:32:43.434432 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.434435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.434452 (+ 17us) server_negotiation.cc:407] Connection header received
0927 16:32:43.434540 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.434542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.434572 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.434632 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.435276 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.435913 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.436819 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.437083 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.437162 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.439339 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.439345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.439351 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.439429 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.441158 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.441590 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.441593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.441595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.441680 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.442019 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.442023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.442025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.442235 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.442403 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.442608 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.442696 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:43.444720 6115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: Bootstrap starting.
I20260927 16:32:43.446942 6115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:43.447829 6115 log.cc:824] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:43.448495 5964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 3
I20260927 16:32:43.450165 6115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: No bootstrap required, opened a new log
I20260927 16:32:43.450443 15734 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 16:32:43.451735 5938 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "709fd63acc5c457d9dec73ac31cc943b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 16:32:43.453120 6039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } attempt: 3
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.190@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:43.454298 6115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.454550 6115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d79ce82957d431a88f5ea45a85882d8, State: Initialized, Role: FOLLOWER
I20260927 16:32:43.454886 6115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.455614 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.455818 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.456254 6115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:43.457741 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.448937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33222)
0927 16:32:43.449048 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.449065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.449151 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.449428 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.449431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.449642 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.449761 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.449771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.450681 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.450684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.451504 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.451516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.451946 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.451950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.452038 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.452506 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.452520 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.453701 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.456010 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.456013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.456015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.456249 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.456626 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.456629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.456631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.456744 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.457232 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.457235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.457338 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.457446 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.457582 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 16:32:43.457741 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.449114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.15.93.188:36847)
0927 16:32:43.449226 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.449230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.449243 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:32:43.449277 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.449279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.449318 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.449394 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.449864 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.450551 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.451614 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.451839 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.451948 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.454068 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.454072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.454075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.454097 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.455821 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.456352 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.456356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.456357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.456487 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.456841 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.456844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.456846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.457133 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.457244 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.457497 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.457611 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 16:32:43 dist-test-slave-grzf krb5kdc[5883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790526763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.93.189@KRBTEST.COM for kudu/127.15.93.188@KRBTEST.COM
I20260927 16:32:43.460899 5964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: Bootstrap starting.
I20260927 16:32:43.460510 6046 master_runner.cc:186] Error getting master registration for 127.15.93.190:36781: OK, instance_id { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" instance_seqno: 1790526762983226 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:32:43.463722 6046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:32:43.464006 5964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:43.464847 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.453865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.15.93.188:36847)
0927 16:32:43.454163 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.454171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.454184 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:32:43.454216 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.454220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.454267 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.455404 (+ 1137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.455516 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.456330 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.457388 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.457735 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.457853 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.460822 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.460826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.460828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.460850 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.462653 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.463193 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.463196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.463198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.463295 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.463660 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.463663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.463664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.463880 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.463973 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.464568 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.464679 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260927 16:32:43.464890 5964 log.cc:824] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:43.465319 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.453461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.93.188:36847 (local address 127.0.0.1:33232)
0927 16:32:43.453583 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 16:32:43.453597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 16:32:43.453680 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 16:32:43.454689 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 16:32:43.454694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 16:32:43.454879 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 16:32:43.455262 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.455271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.456454 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.456457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.457275 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 16:32:43.457285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.457978 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.457985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 16:32:43.458097 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.458595 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 16:32:43.458613 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 16:32:43.460652 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 16:32:43.462794 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.462797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.462799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.463067 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.463428 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 16:32:43.463432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 16:32:43.463434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 16:32:43.463536 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 16:32:43.464045 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 16:32:43.464062 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 16:32:43.464160 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 16:32:43.464264 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 16:32:43.464395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 16:32:43.465583 6133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:43.465646 6133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:43.467413 5964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: No bootstrap required, opened a new log
I20260927 16:32:43.468191 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: Bootstrap starting.
I20260927 16:32:43.470331 5964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.470547 5964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709fd63acc5c457d9dec73ac31cc943b, State: Initialized, Role: FOLLOWER
I20260927 16:32:43.470592 6039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: Neither blocks nor log segments found. Creating new log.
I20260927 16:32:43.470909 5964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.471359 6039 log.cc:824] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: Log is configured to *not* fsync() on all Append() calls
I20260927 16:32:43.472186 5964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:43.472257 5938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 16:32:43.472375 5938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:43.473603 5938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.475185 6089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "709fd63acc5c457d9dec73ac31cc943b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d79ce82957d431a88f5ea45a85882d8"
I20260927 16:32:43.475461 6089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:43.476702 6089 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709fd63acc5c457d9dec73ac31cc943b in term 1.
I20260927 16:32:43.477131 5914 leader_election.cc:302] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d79ce82957d431a88f5ea45a85882d8, 709fd63acc5c457d9dec73ac31cc943b; no voters:
I20260927 16:32:43.477533 6150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 16:32:43.477702 6150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [term 1 LEADER]: Becoming Leader. State: Replica: 709fd63acc5c457d9dec73ac31cc943b, State: Running, Role: LEADER
W20260927 16:32:43.477843 6148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 16:32:43.477938 6148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 16:32:43.478014 6150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
W20260927 16:32:43.479816 5917 tablet.cc:2369] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 16:32:43.479766 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709fd63acc5c457d9dec73ac31cc943b. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.479892 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:43.477756 5896 master_runner.cc:186] Error getting master registration for 127.15.93.189:38159: OK, instance_id { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" instance_seqno: 1790526763199126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 16:32:43.481060 5896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 16:32:43.474512 5938 leader_election.cc:288] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [CANDIDATE]: Term 1 election: Requested vote from peers 4cd315fcdf9c474eb8d2adf83e2b8e68 (127.15.93.189:38159), 2d79ce82957d431a88f5ea45a85882d8 (127.15.93.188:36847)
I20260927 16:32:43.483418 6039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: No bootstrap required, opened a new log
I20260927 16:32:43.483685 6153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 16:32:43.484656 6153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 16:32:43.474499 6136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.489120 6136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.490547 6089 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [term 1 FOLLOWER]: Refusing update from remote peer 709fd63acc5c457d9dec73ac31cc943b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:43.491292 6151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:43.491667 6039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.491971 6039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd315fcdf9c474eb8d2adf83e2b8e68, State: Initialized, Role: FOLLOWER
I20260927 16:32:43.492421 6039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } }
I20260927 16:32:43.494335 6013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "709fd63acc5c457d9dec73ac31cc943b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68"
I20260927 16:32:43.494554 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 16:32:43.495002 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709fd63acc5c457d9dec73ac31cc943b. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.495117 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.496380 6013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 709fd63acc5c457d9dec73ac31cc943b in term 1.
I20260927 16:32:43.496840 6012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [term 1 FOLLOWER]: Refusing update from remote peer 709fd63acc5c457d9dec73ac31cc943b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 16:32:43.497888 6151 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 16:32:43.501384 6151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.501493 6151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:43.502526 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.502631 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.503537 6162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709fd63acc5c457d9dec73ac31cc943b. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.503731 6162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.494277 6039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 16:32:43.504000 6159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.504104 6159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.505822 6136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.505919 6136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b [sys.catalog]: This master's current role is: LEADER
I20260927 16:32:43.505956 6153 catalog_manager.cc:1347] Generated new cluster ID: 89c6860cb30243938352bbaa623aad2f
I20260927 16:32:43.506016 6153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 16:32:43.506987 6159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709fd63acc5c457d9dec73ac31cc943b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709fd63acc5c457d9dec73ac31cc943b" member_type: VOTER last_known_addr { host: "127.15.93.190" port: 36781 } } peers { permanent_uuid: "4cd315fcdf9c474eb8d2adf83e2b8e68" member_type: VOTER last_known_addr { host: "127.15.93.189" port: 38159 } } peers { permanent_uuid: "2d79ce82957d431a88f5ea45a85882d8" member_type: VOTER last_known_addr { host: "127.15.93.188" port: 36847 } } }
I20260927 16:32:43.507086 6159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 16:32:43.515323 6153 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 16:32:43.516444 6153 catalog_manager.cc:1504] Loading token signing keys...
I20260927 16:32:43.517172 6163 mvcc.cc:205] Tried to move back new op lower bound from 7333997623250497536 to 7333997623209394176. Current Snapshot: MvccSnapshot[applied={T|T < 7333997623250497536}]
I20260927 16:32:43.525697 6153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709fd63acc5c457d9dec73ac31cc943b: Generated new TSK 0
I20260927 16:32:43.526388 6153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 16:32:43.530215 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.501857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.15.93.190:36781)
0927 16:32:43.501980 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.501985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.513271 (+ 11286us) server_negotiation.cc:407] Connection header received
0927 16:32:43.513478 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.513484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.513552 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.513608 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.514692 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.515355 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.516599 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.516889 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.516963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.526658 (+ 9695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.526662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.526664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.526691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.528223 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.528744 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.528750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.528755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.528833 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.529166 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.529170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.529171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.529395 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.529504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.529882 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.530029 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:43.535230 6180 catalog_manager.cc:1259] Loaded cluster ID: 89c6860cb30243938352bbaa623aad2f
I20260927 16:32:43.537686 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.502972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.15.93.188:36847)
0927 16:32:43.503072 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.503076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.517660 (+ 14584us) server_negotiation.cc:407] Connection header received
0927 16:32:43.517801 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.517804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.517843 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.517886 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.518705 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.519468 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.520537 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.520871 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.520935 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.528670 (+ 7735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.528677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.528682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.528711 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.530346 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.535529 (+ 5183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.535533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.535536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.535642 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.536684 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.536688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.536689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.536913 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.537035 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.537371 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.537487 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:32:43.537436 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.502271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.15.93.189:38159)
0927 16:32:43.502364 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.502368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.521483 (+ 19115us) server_negotiation.cc:407] Connection header received
0927 16:32:43.521648 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.521651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.521694 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.521740 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.522595 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.523237 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.524268 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.524566 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.524641 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.530605 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 16:32:43.530609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 16:32:43.530612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 16:32:43.530642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 16:32:43.532168 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.535784 (+ 3616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.535789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.535790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.535883 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 16:32:43.536333 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 16:32:43.536339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 16:32:43.536342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 16:32:43.536585 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 16:32:43.536705 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.537155 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.537283 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 16:32:43.539534 6180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: loading cluster ID for follower catalog manager: success
I20260927 16:32:43.541944 6180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: acquiring CA information for follower catalog manager: success
I20260927 16:32:43.543160 6180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4cd315fcdf9c474eb8d2adf83e2b8e68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 16:32:43.566916 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.563635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.15.93.188:36847)
0927 16:32:43.563757 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.563762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.565482 (+ 1720us) server_negotiation.cc:407] Connection header received
0927 16:32:43.565985 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.565990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.566046 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.566099 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.566781 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33260: BlockingRecv error: recv got EOF from 127.0.0.1:33260 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 16:32:43.568362 5968 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.564460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.15.93.190:36781)
0927 16:32:43.564586 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.564590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.567425 (+ 2835us) server_negotiation.cc:407] Connection header received
0927 16:32:43.567663 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.567666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.567715 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.567763 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.568236 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56006: BlockingRecv error: recv got EOF from 127.0.0.1:56006 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 16:32:43.569646 6045 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.565102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.15.93.189:38159)
0927 16:32:43.565215 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.565233 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.568739 (+ 3506us) server_negotiation.cc:407] Connection header received
0927 16:32:43.568978 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.568981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.569029 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.569079 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.569538 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49310: BlockingRecv error: recv got EOF from 127.0.0.1:49310 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 16:32:43.577898 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.572004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.15.93.189:38159)
0927 16:32:43.572110 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.572114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.573151 (+ 1037us) server_negotiation.cc:407] Connection header received
0927 16:32:43.573378 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.573381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.573536 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.573618 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.573977 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.574783 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.576088 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.576427 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.576494 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.576919 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.577444 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.577546 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.577725 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.577776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 16:32:43.583284 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.581781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.15.93.188:36847)
0927 16:32:43.581885 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.581889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.582334 (+ 445us) server_negotiation.cc:407] Connection header received
0927 16:32:43.582553 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.582556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.582631 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.582685 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.583182 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33274: BlockingRecv error: recv got EOF from 127.0.0.1:33274 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 16:32:43.590899 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.586620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.15.93.189:38159)
0927 16:32:43.586713 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.586717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.587126 (+ 409us) server_negotiation.cc:407] Connection header received
0927 16:32:43.587290 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.587293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.587335 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.587406 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.587726 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.588524 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.589723 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.590049 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.590115 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.590446 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.590537 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.590581 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.590740 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.590784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 16:32:43.596210 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.594787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.15.93.188:36847)
0927 16:32:43.594881 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.594885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.595349 (+ 464us) server_negotiation.cc:407] Connection header received
0927 16:32:43.595554 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.595557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.595605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.595658 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.596116 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33284: BlockingRecv error: recv got EOF from 127.0.0.1:33284 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:43.604673 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.600168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.15.93.189:38159)
0927 16:32:43.600259 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.600263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.600720 (+ 457us) server_negotiation.cc:407] Connection header received
0927 16:32:43.600886 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.600889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.600931 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.600972 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.601285 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.602037 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.603284 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.603653 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.603721 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.604076 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.604171 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.604215 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.604502 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.604556 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 16:32:43.609831 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.608448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.15.93.188:36847)
0927 16:32:43.608555 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.608559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.608992 (+ 433us) server_negotiation.cc:407] Connection header received
0927 16:32:43.609164 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.609168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.609214 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.609262 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.609736 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33294: BlockingRecv error: recv got EOF from 127.0.0.1:33294 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:43.620010 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.615646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.15.93.189:38159)
0927 16:32:43.615789 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.615793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.616216 (+ 423us) server_negotiation.cc:407] Connection header received
0927 16:32:43.616388 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.616391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.616437 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.616481 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.616821 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.617592 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.618800 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.619127 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.619195 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.619569 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.619647 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.619689 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.619842 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.619888 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 16:32:43.625108 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.623665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.15.93.188:36847)
0927 16:32:43.623793 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.623798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.624263 (+ 465us) server_negotiation.cc:407] Connection header received
0927 16:32:43.624432 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.624435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.624484 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.624545 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.625012 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33296: BlockingRecv error: recv got EOF from 127.0.0.1:33296 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 16:32:43.639999 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.635363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.15.93.189:38159)
0927 16:32:43.635537 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.635543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.636040 (+ 497us) server_negotiation.cc:407] Connection header received
0927 16:32:43.636281 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.636285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.636338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.636384 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.636733 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.637515 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.638748 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.639078 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.639147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.639528 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.639609 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.639652 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.639811 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.639860 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:32:43.645524 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.644086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.15.93.188:36847)
0927 16:32:43.644195 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.644199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.644609 (+ 410us) server_negotiation.cc:407] Connection header received
0927 16:32:43.644791 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.644794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.644846 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.644896 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.645397 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33310: BlockingRecv error: recv got EOF from 127.0.0.1:33310 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 16:32:43.668972 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.664035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.15.93.189:38159)
0927 16:32:43.664168 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.664173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.664760 (+ 587us) server_negotiation.cc:407] Connection header received
0927 16:32:43.664967 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.664970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.665024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.665073 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.665470 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.666250 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.667570 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.667899 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.667990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.668450 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.668552 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.668604 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.668780 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.668834 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 16:32:43.674671 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.673165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.15.93.188:36847)
0927 16:32:43.673275 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.673279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.673702 (+ 423us) server_negotiation.cc:407] Connection header received
0927 16:32:43.673883 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.673886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.673947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.673998 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.674555 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33320: BlockingRecv error: recv got EOF from 127.0.0.1:33320 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 16:32:43.713073 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.709325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.15.93.189:38159)
0927 16:32:43.709471 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.709476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.709661 (+ 185us) server_negotiation.cc:407] Connection header received
0927 16:32:43.709777 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.709780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.709835 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.709939 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.710233 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.711058 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.711968 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.712313 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.712414 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.712634 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.712721 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.712792 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.712870 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.712925 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 16:32:43.717414 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.716339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.15.93.188:36847)
0927 16:32:43.716461 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.716466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.716662 (+ 196us) server_negotiation.cc:407] Connection header received
0927 16:32:43.716785 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.716787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.716831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.716896 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.717324 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33332: BlockingRecv error: recv got EOF from 127.0.0.1:33332 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 16:32:43.786885 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.783282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.15.93.189:38159)
0927 16:32:43.783446 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.783449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.783621 (+ 172us) server_negotiation.cc:407] Connection header received
0927 16:32:43.783747 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.783750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.783794 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.783871 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.784158 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.784735 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.785753 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.786003 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.786121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.786386 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.786463 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.786553 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.786679 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.786732 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 16:32:43.790871 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.789725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.15.93.188:36847)
0927 16:32:43.789867 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.789872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.789993 (+ 121us) server_negotiation.cc:407] Connection header received
0927 16:32:43.790154 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.790157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.790216 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.790311 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.790772 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33340: BlockingRecv error: recv got EOF from 127.0.0.1:33340 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 16:32:43.923782 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:43.920892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.15.93.189:38159)
0927 16:32:43.921059 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.921063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.921164 (+ 101us) server_negotiation.cc:407] Connection header received
0927 16:32:43.921283 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.921286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.921333 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.921402 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:43.921605 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.922095 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.922876 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:43.923078 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:43.923148 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:43.923348 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:43.923432 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:43.923483 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:43.923562 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 16:32:43.923608 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 16:32:43.927335 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:43.926358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.15.93.188:36847)
0927 16:32:43.926499 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:43.926502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:43.926516 (+ 14us) server_negotiation.cc:407] Connection header received
0927 16:32:43.926638 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:43.926640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:43.926682 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:43.926747 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:43.927227 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33354: BlockingRecv error: recv got EOF from 127.0.0.1:33354 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 16:32:44.188457 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.185333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.15.93.189:38159)
0927 16:32:44.185489 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.185494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.185617 (+ 123us) server_negotiation.cc:407] Connection header received
0927 16:32:44.185792 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.185801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.185844 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.185948 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.186219 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.186721 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.187551 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.187761 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.187840 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.188048 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.188112 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.188163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.188254 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.188295 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 16:32:44.192396 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 16:32:44.191300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.15.93.188:36847)
0927 16:32:44.191453 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.191457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.191606 (+ 149us) server_negotiation.cc:407] Connection header received
0927 16:32:44.191751 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.191754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.191805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.191888 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 16:32:44.192304 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33366: BlockingRecv error: recv got EOF from 127.0.0.1:33366 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 16:32:44.466974 6133 catalog_manager.cc:1259] Loaded cluster ID: 89c6860cb30243938352bbaa623aad2f
I20260927 16:32:44.467056 6133 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: loading cluster ID for follower catalog manager: success
I20260927 16:32:44.468659 6133 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: acquiring CA information for follower catalog manager: success
I20260927 16:32:44.469708 6133 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d79ce82957d431a88f5ea45a85882d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 16:32:44.709486 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.706417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.15.93.189:38159)
0927 16:32:44.706660 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.706664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.706680 (+ 16us) server_negotiation.cc:407] Connection header received
0927 16:32:44.706799 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.706802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.706849 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.706925 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.707136 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.707720 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.708500 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.708716 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.708787 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.708983 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.709065 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.709115 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.709281 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.709332 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260927 16:32:44.715466 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.712173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.15.93.188:36847)
0927 16:32:44.712394 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.712397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.712410 (+ 13us) server_negotiation.cc:407] Connection header received
0927 16:32:44.712464 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.712466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.712596 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.712687 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.712865 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.713360 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.714098 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.714312 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.714384 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.714634 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.715047 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.715138 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.715195 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.715239 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260927 16:32:44.722775 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.718219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.15.93.188:36847)
0927 16:32:44.718318 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.718325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.718509 (+ 184us) server_negotiation.cc:407] Connection header received
0927 16:32:44.718623 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.718626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.718675 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.718749 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.719074 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.719813 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.720839 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.721052 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.721161 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.721892 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.721974 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.722041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.722581 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.722643 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 16:32:44.722822 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.718664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.15.93.189:38159)
0927 16:32:44.718793 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.718797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.719103 (+ 306us) server_negotiation.cc:407] Connection header received
0927 16:32:44.719201 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.719204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.719252 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.719320 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.719670 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.720382 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.721469 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.721721 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.721816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.722342 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.722416 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.722479 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.722641 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.722699 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 16:32:44.723049 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 16:32:44.718270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.15.93.190:36781)
0927 16:32:44.718523 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 16:32:44.718527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 16:32:44.718877 (+ 350us) server_negotiation.cc:407] Connection header received
0927 16:32:44.719001 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 16:32:44.719004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 16:32:44.719178 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 16:32:44.719290 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 16:32:44.719533 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.720195 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.721356 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 16:32:44.721629 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 16:32:44.721753 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 16:32:44.722082 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 16:32:44.722431 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 16:32:44.722529 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 16:32:44.722802 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 16:32:44.722874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260927 16:32:44.726964 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5896
I20260927 16:32:44.733474 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 5971
I20260927 16:32:44.739523 15734 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Uwdq0/build/debug/bin/kudu with pid 6046
2026-09-27T16:32:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3067 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6123 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276597 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST